Auxiliary Views: Why Data Diversity Helps, and What Repetition Is Actually For

Paper: 2609.04180 Authors: Joseph Lee, Yidi Huang, Dokyoon Kim, Shu Yang, Li Shen Categories: cs.AI, cs.CL

The Gap

Large pre-training corpora are built by deciding how to spend tokens, and one decision dominates: how much to repeat a document. Repetition demonstrably helps — that much is established — but why it helps and what it is doing are not, and neither is what the alternative could be.

The paper’s hypothesis is specific: auxiliary views — reformulations of knowledge — are causally helpful for learning. If true, it reframes repetition: repeating a document is one way to give the model more exposure to a piece of knowledge, and reformulating it is another. Those two are different in kind, and a token budget forces a choice between them. So the question becomes compressible and testable: at a fixed budget, which exposure buys more?

Testing it requires controlling the confounds, because “paraphrases help” is confounded with “more tokens help” and with “a stronger teacher helps”. The paper’s contribution includes separating those.

   PRE-TRAINING CORPORA: A DOMINANT ALLOCATION DECISION

   how much to REPEAT a document
        |
        v
   repetition DEMONSTRABLY HELPS -- that much is established
   but WHY it helps, and WHAT IT IS DOING, are not
   and neither is WHAT THE ALTERNATIVE COULD BE
        |
        v
   [THE PAPER'S HYPOTHESIS]
     AUXILIARY VIEWS -- REFORMULATIONS of knowledge --
     are CAUSALLY HELPFUL for learning
        |
        v
   IF TRUE, IT REFRAMES REPETITION
     repetition     = ONE way to give the model more exposure
                      to a piece of knowledge
     reformulation  = ANOTHER
     -> different IN KIND, and a token budget FORCES A CHOICE
        |
        v
   [TESTABLE QUESTION] AT A FIXED BUDGET, WHICH EXPOSURE
                       BUYS MORE?
        |
        v
   AND the confounds must be controlled:
     "paraphrases help" is confounded with "MORE TOKENS help"
     and with "A STRONGER TEACHER helps"

The Increment

One sentence: Before this paper, data diversity was known to matter without a mechanism; after it, controlled experiments show that moving tokens from repetition to auxiliary views improves learning, that the benefit survives weak teachers, and that it depends on the kind of knowledge being reformulated.

Core Mechanism

The design is a set of controlled experiments, and the findings come in a deliberate order — each one closing a confound the previous one leaves open.

Repetition is necessary for acquisition, and paraphrasing helps only at smaller batch sizes. The first finding grants repetition its established role, and the batch-size qualifier is the interesting part: it locates paraphrasing’s benefit in a regime rather than claiming it universally. Anyone who has watched paraphrase data fail at scale now has a reason.

Holding the token budget fixed, allocating tokens from document repetition to auxiliary views improves learning — and then the clause that makes this more than a budget reshuffle: counterintuitively, even for factual recall. That is the sharp result. The intuition is that reformulations help with generalisation and repetition helps with memorisation, so a memorisation task should prefer repetition. It does not. So the auxiliary views are not merely a generalisation aid; they are a better way to spend tokens for acquisition too, which is a stronger and less expected claim.

The effectiveness of auxiliary views is not contingent on the strength of the teacher model that generates them. This is the confound-closing finding, and it is the one that makes the mechanism credible. If better paraphrases from stronger models helped more, the effect would be explainable as “a good teacher taught the model” — distillation by another name. Since weak teachers suffice, the benefit comes from the existence and diversity of reformulation rather than from the quality of any particular one.

The paper identifies forms of knowledge — contextual and foundational — that aid learning in the presence of prior knowledge gaps. So the benefit is not uniform across content: some kinds of auxiliary view help precisely where the model lacks background, which is a practically useful targeting rule rather than a general claim that more variety is better.

Mechanistically, the effects manifest via layer-wise biases and compression — connecting the corpus-level manipulation to internal structure rather than stopping at the benchmark delta.

The paper’s conclusion ties it back to the motivating phenomenon: auxiliary representations of knowledge, which arise naturally in large pre-training corpora, are a key factor in the success of pre-training, and this offers a plausible explanation for why data diversity matters. The phrase “arise naturally” is doing real work — it suggests the mechanism is already operating in existing corpora through incidental reformulation, which is a more satisfying explanation than “diversity is good” and it predicts that deliberately increasing reformulation should keep helping.

   FINDINGS, IN THE ORDER THEY CLOSE CONFOUNDS

   [1] REPETITION IS NECESSARY for acquisition
       AND PARAPHRASING HELPS ONLY AT SMALLER BATCH SIZES
         <- grants repetition its established role
         <- locates paraphrasing's benefit IN A REGIME rather
            than claiming it universally
         -> explains why paraphrase data can fail AT SCALE

   [2] TOKEN BUDGET FIXED: moving tokens from document
       repetition to AUXILIARY VIEWS IMPROVES LEARNING
         AND -- counterintuitively -- EVEN FOR FACTUAL RECALL
         <- the SHARP result
         <- intuition says reformulations aid GENERALISATION and
            repetition aids MEMORISATION, so a memorisation task
            should prefer repetition. IT DOES NOT.
         -> auxiliary views are not merely a generalisation aid;
            they are a better way to spend tokens for ACQUISITION

   [3] EFFECTIVENESS IS NOT CONTINGENT ON THE TEACHER'S STRENGTH
         <- the confound-closing finding that makes the mechanism
            credible
         <- if better paraphrases from stronger models helped more,
            the effect would reduce to "a good teacher taught it"
            (distillation by another name)
         -> the benefit comes from the EXISTENCE AND DIVERSITY of
            reformulation, not the QUALITY of any particular one

   [4] FORMS OF KNOWLEDGE MATTER
         CONTEXTUAL and FOUNDATIONAL forms aid learning in the
         presence of PRIOR KNOWLEDGE GAPS
         -> not uniform across content: a TARGETING RULE rather
            than "more variety is better"

   [5] MECHANISM: effects manifest via LAYER-WISE BIASES and
       COMPRESSION
         -> connects the corpus-level manipulation to internal
            structure, rather than stopping at the benchmark delta

   CONCLUSION
     auxiliary representations of knowledge, which ARISE NATURALLY
     in large pre-training corpora, are a KEY FACTOR in pre-training
     success -> a plausible explanation for WHY DATA DIVERSITY MATTERS
       <- "arise naturally" matters: the mechanism is already
          operating in existing corpora through INCIDENTAL
          reformulation, and deliberately increasing it should
          keep helping

Think of it as studying a language by reading the same page five times versus reading five different explanations of the same material. Repetition works — everyone who has crammed knows it. But if you have a fixed number of hours, spending some of them on a different author’s framing of the same idea turns out to serve even memorisation better, which is not what the cramming intuition predicts. And the strongest part: it does not matter whether the alternative explanations come from a brilliant teacher or a mediocre one. What matters is that the material is reframed, because the reframing is what makes the structure visible rather than the surface. That also explains why the effect is uneven — reframing a fact you have no context for helps most, while reframing something already understood adds less.

Key Concepts

• Auxiliary views as reformulations: different presentations of the same knowledge. Treating them as a distinct kind of exposure — rather than as a variant of repetition — is what turns the token budget into a choice worth measuring.
• Fixed-budget comparison: holding tokens constant so the comparison is between kinds of exposure rather than amounts. Without it, “paraphrases help” is indistinguishable from “more data helps”.
• The factual-recall result as the sharp finding: auxiliary views beat repetition even where memorisation is the goal. It rules out the clean division of labour the intuition suggests.
• Teacher-independence as the credibility check: the benefit survives weak teachers, so the effect is the diversity of reformulation rather than the quality of a particular teacher. This is what separates it from distillation.
• Contextual and foundational knowledge as targets: the benefit concentrates where prior knowledge is missing, which is a rule for where to spend reformulation effort.

Framework Shift

Before (diversity as a known good):
  repeat documents to expose the model to knowledge
  -> repetition helps, unexplained
  -> "data diversity matters" as an observation
  -> paraphrases either trusted or distrusted wholesale,
     with no account of when they help

After (reformulation as a distinct use of tokens):
  auxiliary views are causally helpful; repetition is one way to
    give exposure and reformulation is another
  -> at a FIXED budget, auxiliary views win, even for factual recall
  -> weak teachers suffice: the benefit is DIVERSITY, not quality
  -> targeted at context gaps, with layer-wise mechanism

From knowing that diversity helps without knowing why, to identifying reformulation as a distinct and budget-competing form of exposure whose benefit is teacher-independent, the core shift is that repetition and reformulation are substitutes for a shared purpose — and the substitute is better.

Expert Assessment

Problem choice: Excellent, and it asks a question the field has been spending enormous resources on without a mechanism. Pre-training data mixtures are chosen at scale, repetition budgets are set deliberately, and “diversity matters” has been an observation rather than a causal account.

Method maturity: The design’s strength is the sequence of confound closures. Granting repetition its role, then showing the fixed-budget reallocation helps, then removing the teacher-quality explanation, then localising the effect by knowledge type, and finally connecting to internal structure — each step would have been a paper on its own, and the ordering is what makes the mechanism credible rather than merely plausible. The batch-size qualifier on paraphrasing is the kind of finding that saves practitioners from a costly mistake.

Experimental integrity: The teacher-independence result is the most valuable control, because it is exactly the alternative explanation a sceptic would offer, and it is reported in the direction that weakens the paper’s own framing (the teacher is not doing the work). The factual-recall result cuts against intuition and is therefore more informative. The main limitation is that these are controlled experiments, so the reformulations are generated under the study’s own conditions; how much incidental reformulation exists in real corpora, and whether deliberately adding more keeps paying, are extrapolations the paper offers as plausible rather than demonstrated.

Writing quality: The findings are ordered as confound closures, which makes the paper easy to follow and each result easy to place. Since the practical decision is a token-allocation rule, a short passage translating the findings into guidance — how much of a repetition budget to convert, and for which kinds of content — would make it directly actionable for anyone assembling a pre-training mixture.

Verdict: strong accept — it supplies a causal account for a phenomenon the field has been allocating resources by, pins the mechanism by removing both obvious confounds, and locates the benefit by knowledge type.

Takeaways

• Treat reformulation as a distinct use of tokens, competing with repetition. At a fixed budget, the two are substitutes, and the substitute measured better here.
• Check whether your effect is really teacher quality. If the benefit survives weak teachers, the mechanism is diversity rather than the quality of any particular example.
• Do not assume memorisation prefers repetition. The result held even for factual recall, which eliminates the clean division of labour intuition suggests.
• Target reformulation where context is missing. The benefit concentrated on contextual and foundational material the model lacked background for.