MoEs Overfit More to Repeated Data, and Sparsity Is What Drives It

Paper: 2609.11917 Authors: Atindra Jha, Margaret Li, Jure Leskovec, Percy Liang, Luke Zettlemoyer Categories: cs.CL, cs.LG

The Gap

As the supply of human-written text is exhausted, repeating language model training data has become standard practice. So the properties of repetition are not a niche question; they are a property of how models are trained now.

And the question had been answered for the wrong architecture. Prior work studied data repetition for densely activated Transformers, while the effects of data repetition remain largely unexplored for recently dominant sparse architectures such as Mixture-of-Experts — despite their increased compute efficiency. That combination is what makes the gap consequential: MoE’s whole selling point is compute efficiency per unit of quality, and its efficiency comes from activating a subset of parameters per token. If sparse activation interacts badly with repeated data, then the architecture people adopt for efficiency has an unexamined dependence on an increasingly standard training practice.

   REPETITION IS STANDARD, AND ITS EFFECTS WERE STUDIED
   FOR THE WRONG ARCHITECTURE

   supply of human-written text EXHAUSTED
     -> REPEATING TRAINING DATA IS STANDARD PRACTICE
     -> so repetition's properties are not a NICHE question;
        they are a property of HOW MODELS ARE TRAINED NOW
        |
        v
   AND THE QUESTION WAS ANSWERED FOR THE WRONG ARCHITECTURE
     prior work: data repetition for DENSELY ACTIVATED TRANSFORMERS
     largely unexplored: RECENTLY DOMINANT SPARSE ARCHITECTURES such
       as MIXTURE-OF-EXPERTS -- DESPITE their increased compute
       efficiency
        |
        v
   [WHY THAT COMBINATION MAKES THE GAP CONSEQUENTIAL]
     MoE's whole selling point is COMPUTE EFFICIENCY PER UNIT OF
       QUALITY
     its efficiency comes from ACTIVATING A SUBSET OF PARAMETERS
       PER TOKEN
        |
        v
   -> if SPARSE ACTIVATION interacts badly with REPEATED DATA,
      then the architecture people adopt for EFFICIENCY has an
      UNEXAMINED DEPENDENCE on an INCREASINGLY STANDARD training
      practice

The Increment

One sentence: Before this paper, repetition’s effect on Mixture-of-Experts was unknown; after it, MoEs degrade faster, the effect scales with sparsity via total rather than active parameters, and masking-based regularization recovers the advantage at heavy repetition.

Core Mechanism

The sweep is designed to separate the variables that a single comparison would conflate: repetition rates varied across single- and multi-domain data mixes, and across MoE settings including expert count and granularity. Three axes at once — how much repetition, what the data is, and how the sparse architecture is configured. That breadth is what allows the findings to be attributed rather than merely observed.

The headline result is a comparison across architecture classes at matched scale: for models ranging from 80M to 1B active (8.5B total) parameters, MoEs degrade more rapidly under data repetition. Two things about that sentence matter. The range spans more than an order of magnitude, so the effect is not a small-model artefact. And the phrasing “active (total)” is deliberate — the models are matched on active parameters while differing in total, which is a fair comparison for inference cost but is also where the mechanism finding comes from.

That mechanism finding is the sharpest part: the effect increases with sparsity, dictated by total rather than active parameters. So the relevant quantity is how many parameters exist, not how many fire per token. That is counterintuitive and it is exactly what the architecture’s premise would not predict: MoE efficiency is usually reasoned about in terms of active parameters, and here the fragility tracks the inactive ones. The interpretation is that the extra parameters are where the memorisation lives.

The numbers give the shape of the degradation and of the trade:

• 80M dense models can repeat data over 8× with minimal degradation.
• MoEs instead begin to suffer at 4×, and deteriorate rapidly.
• They cede their performance benefits in all-unique data settings to underperform dense models after 32×.

Read together, those three lines describe a crossover: the sparse architecture wins when data is unique and loses when it is heavily repeated, with the reversal somewhere between 4× and 32× depending on how severe the degradation becomes.

Then the remedy, reported with its own limit. Some existing regularization methods can mitigate overfitting — dropout in particular — and with strong masking-based regularization, MoEs are able to outperform dense models even when data is repeated more than 64 times. But: no method fully matches the performance of all-unique training data. So regularization shifts the crossover rather than removing the penalty, which is the honest form of a mitigation result.

And the mechanism analysis connects the behaviour to internals: MoE routing universally stabilizes early in training, and expert specialization correlates with overfitting to repeated data. Two findings, and the second is the load-bearing one: overfitting is associated with experts specialising, which suggests the failure is not generic capacity excess but a specific division of labour that repeated data encourages. That points the suggested future work in a direction — reducing over-specialization by disrupting memorization patterns — rather than merely asking for better regularizers.

   THE SWEEP IS DESIGNED TO SEPARATE VARIABLES THAT A SINGLE COMPARISON
   WOULD CONFLATE
     repetition rates varied across SINGLE- and MULTI-DOMAIN mixes
     AND across MoE settings including EXPERT COUNT and GRANULARITY
       <- THREE AXES AT ONCE: how much repetition, what the data is,
          and how the sparse architecture is configured
       -> breadth allows findings to be ATTRIBUTED rather than merely
          OBSERVED

   THE HEADLINE RESULT: A COMPARISON ACROSS ARCHITECTURE CLASSES AT
   MATCHED SCALE
     for models ranging from 80M to 1B ACTIVE (8.5B TOTAL) parameters,
     MOES DEGRADE MORE RAPIDLY UNDER DATA REPETITION
       <- the range spans MORE THAN AN ORDER OF MAGNITUDE
          -> not a SMALL-MODEL ARTEFACT
       <- "ACTIVE (TOTAL)" is DELIBERATE: matched on ACTIVE parameters
          while differing in TOTAL
          -> fair for INFERENCE COST, and also WHERE THE MECHANISM
             FINDING COMES FROM

   THE MECHANISM FINDING IS THE SHARPEST PART
     the effect INCREASES WITH SPARSITY, DICTATED BY TOTAL RATHER THAN
     ACTIVE PARAMETERS
       -> the relevant quantity is HOW MANY PARAMETERS EXIST, not how
          many fire per token
       <- COUNTERINTUITIVE, and exactly what the architecture's premise
          would NOT predict: MoE efficiency is usually reasoned about in
          terms of ACTIVE parameters, and here the fragility tracks the
          INACTIVE ONES
       -> interpretation: the EXTRA PARAMETERS are where the
          MEMORISATION LIVES

   THE NUMBERS GIVE THE SHAPE OF THE DEGRADATION AND OF THE TRADE
     80M DENSE models can repeat data over 8x with MINIMAL degradation
     MOEs instead begin to suffer at 4x, and DETERIORATE RAPIDLY
     they CEDE their performance benefits in all-unique data settings to
       UNDERPERFORM DENSE MODELS after 32x
     <- read together, these describe a CROSSOVER: the sparse
        architecture WINS on unique data and LOSES on heavily repeated
        data, with the reversal somewhere between 4x and 32x

   THEN THE REMEDY, REPORTED WITH ITS OWN LIMIT
     some existing regularization can MITIGATE OVERFITTING -- DROPOUT
       in particular
     with STRONG MASKING-BASED REGULARIZATION, MoEs OUTPERFORM DENSE
       MODELS even when data is repeated MORE THAN 64 TIMES
     BUT: NO METHOD FULLY MATCHES THE PERFORMANCE OF ALL-UNIQUE
       TRAINING DATA
       <- regularization SHIFTS THE CROSSOVER rather than REMOVING THE
          PENALTY: the honest form of a mitigation result

   AND THE MECHANISM ANALYSIS CONNECTS BEHAVIOUR TO INTERNALS
     MoE ROUTING UNIVERSALLY STABILIZES EARLY in training
     EXPERT SPECIALIZATION CORRELATES WITH OVERFITTING TO REPEATED DATA
       <- the LOAD-BEARING one: overfitting is associated with experts
          SPECIALISING
       -> suggests the failure is NOT GENERIC CAPACITY EXCESS but a
          SPECIFIC DIVISION OF LABOUR that repeated data ENCOURAGES
       -> points future work in a DIRECTION -- REDUCING
          OVER-SPECIALIZATION BY DISRUPTING MEMORIZATION PATTERNS --
          rather than merely asking for BETTER REGULARIZERS

Think of it as a large team where each member becomes an expert in one specific customer’s quirks. With a diverse customer base that is valuable: each specialist handles their cases well. Repeat the same few customers over and over and the specialisation hardens around their idiosyncrasies — the team gets better at those exact cases and worse at everything else, which is not what more practice should do. That is the shape of the finding: the fragility grows with the number of specialists asked for (total parameters), not with how many are active on any given case, because the specialisation itself is where the memorisation is stored. And it explains why regularization only shifts the crossover: dropout and masking spread the load, but they do not stop specialists from forming.

Key Concepts

• Repetition as a standard practice with an unexamined interaction: the effect was characterised for dense models and not for sparse ones. The consequence is that the efficiency-motivated architecture depends on a practice nobody checked it against.
• Degradation tracking total rather than active parameters: fragility scales with how many parameters exist, not how many fire. It inverts the quantity MoE reasoning usually centres on.
• The crossover between architecture classes: dense wins under heavy repetition, sparse wins on unique data, with reversal between 4× and 32×. It makes the choice conditional on the data regime rather than absolute.
• Regularization that shifts rather than removes: masking-based methods let MoEs beat dense models at over 64× repetition, without matching all-unique training. The remaining gap is the honest part of the remedy.
• Specialization as the correlate of overfitting: expert specialisation rises with repeated data. It locates the failure in a division of labour rather than in raw capacity.

Framework Shift

Before (repetition characterised for dense models):
  repeat data as human text runs out
  -> the effect is known for dense transformers
  -> sparse architectures assumed to behave comparably
  -> no account of the interaction with sparsity

After (the interaction measured, cross-class):
  repetition rates swept across mixes and MoE configurations
  -> MoEs degrade faster (80M-1B active, up to 8.5B total)
  -> the effect scales with TOTAL parameters, not active
  -> dense wins under heavy repetition; sparse wins on unique data
  -> masking-based regularization shifts the crossover but does not
     remove the penalty
  -> expert specialization correlates with the overfitting

From assuming that an efficiency-motivated architecture inherits the known properties of repetition, to measuring a crossover between architecture classes and locating it in expert specialisation, the core shift is that sparsity changes how a model responds to repeated data, and it does so through inactive parameters.

Expert Assessment

Problem choice: Excellent, and the practical framing is what makes it matter. Repetition is already standard rather than hypothetical, and the architecture being tested is the one chosen for efficiency — so an unexamined interaction between the two is exactly the kind of dependency worth finding before it is relied on.

Method maturity: The sweep across three axes — repetition level, data mix, and MoE configuration including expert count and granularity — is what permits attribution rather than description. The scale range from 80M to 8.5B total rules out a small-model explanation. And the mechanism analysis is what elevates the paper above a scaling study: finding that routing stabilizes early and that expert specialisation correlates with the overfitting gives the result a cause, and it points the suggested remedy at over-specialisation rather than at regularisation in general.

Experimental integrity: The comparison is honest in a subtle way: models are matched on active parameters, which is the fair basis for inference cost, and the paper then reports that the fragility tracks total parameters, which is the axis on which they are not matched. That is a finding that cuts against the convenience of the setup, and reporting it is what makes the mechanism claim credible. The mitigation result is also reported with its limit intact — no method matches all-unique data — rather than being presented as a solution.

Writing quality: The numbers are arranged to reveal the crossover rather than to show a single degradation figure, which is the right presentation for a trade-off. Because the practical reader wants to know what to do, a short passage translating the mechanism into a diagnostic — for instance whether monitoring expert specialisation could serve as an early warning — would make the finding actionable rather than merely explanatory.

Verdict: strong accept — it measures an unexamined interaction between a standard training practice and the architecture chosen for efficiency, traces the fragility to inactive parameters, and locates the mechanism in expert specialisation.

Takeaways

• Check interactions, not just components. Repetition was characterised for dense models, and the architecture adopted for efficiency inherits an unverified assumption.
• Watch which parameter count a property tracks. Fragility following total rather than active parameters inverts the quantity that sparsity is usually justified by.
• Present trade-offs as crossovers. Dense versus sparse reverses somewhere between 4× and 32× repetition, which makes the choice a regime question rather than a verdict.
• Look for the specialisation behind overfitting. If experts hardening on repeated data is the mechanism, spreading the load is not the same fix as preventing the hardening.