Data Repetition Beats Data Scaling in Long-CoT Supervised Fine-Tuning

Paper: 2602.11149 Authors: Dawid J. Kopiczko, Sagar Vaze, Tijmen Blankevoort, Yuki M. Asano Categories: cs.CL

Abstract

This paper challenges the conventional machine learning wisdom that more unique training samples lead to better generalization. The authors demonstrate that supervised fine-tuning (SFT) for chain-of-thought reasoning benefits significantly from data repetition. Under a fixed computational budget, training for multiple epochs on smaller datasets substantially outperforms single-epoch training on larger datasets. Using Olmo3-7B on AIME’24/25 and GPQA benchmarks, they show that 128 epochs on 400 samples outperforms 1 epoch on 51,200 samples by 12-26 percentage points, without catastrophic forgetting. The study identifies training token accuracy as a reliable signal for when repetition saturates, with improvements plateauing at full memorization.

Key Contributions

• Counterintuitive finding: Multi-epoch training on small datasets outperforms single-epoch training on proportionally larger datasets under fixed update budgets
• Empirical validation: Demonstrated 12-26 percentage point improvements on challenging reasoning benchmarks (AIME’24/25, GPQA)
• Practical stopping criterion: Training token accuracy reliably signals saturation, providing a cost-effective alternative to expensive data scaling
• No catastrophic forgetting: Extensive repetition does not degrade performance on held-out evaluation sets
• New research direction: Poses the “repetition advantage” as an open problem for understanding LLM training dynamics

Methodology and Experimental Design

The researchers conducted systematic experiments controlling for the total number of gradient updates while varying the ratio of dataset size to training epochs. They used Olmo3-7B as their base model and focused on chain-of-thought reasoning tasks requiring multi-step mathematical and scientific reasoning.

The experimental setup maintained a constant update budget: training for $E$ epochs on $N$ samples equals $E \times N$ total sample presentations. For instance, 128 epochs on 400 samples ($51,200$ total presentations) was compared against 1 epoch on 51,200 unique samples. This design isolates the effect of repetition from computational cost.

The authors evaluated on two challenging benchmarks:

• AIME (American Invitational Mathematics Examination): High-difficulty mathematical problems requiring sophisticated reasoning
• GPQA (Graduate-Level Google-Proof Q&A): Expert-level science questions designed to be difficult even for search-assisted humans

Training token accuracy—the model’s accuracy on predicting the next token during training—served as the primary monitoring metric for determining when additional epochs cease to provide benefits.

Results and Analysis

The results reveal a striking pattern: repetition consistently outperforms diversity under fixed budgets. On AIME’24, the 128-epoch configuration achieved 26 percentage points higher accuracy than the single-epoch baseline. On GPQA, the improvement was 12 percentage points. These gains are substantial given the difficulty of these benchmarks.

The relationship between training token accuracy and generalization performance follows a predictable trajectory. As models approach 100% training token accuracy (full memorization), test performance improvements plateau. This saturation point occurs consistently across different dataset sizes and model configurations, suggesting a fundamental property of reasoning SFT rather than an artifact of specific experimental choices.

Critically, the authors found no evidence of catastrophic forgetting even after 128 epochs of training. Models maintained their performance on held-out evaluation sets, indicating that memorization of training data does not come at the cost of general reasoning capability. This finding contradicts common concerns about overfitting in deep learning.

The computational efficiency implications are significant. Rather than collecting and curating tens of thousands of unique chain-of-thought examples, practitioners can achieve superior results with hundreds of high-quality examples trained for many epochs. This dramatically reduces data collection costs while improving model performance.

Implications for LLM Training

This work has profound implications for how we approach supervised fine-tuning of reasoning models. The traditional paradigm emphasizes scaling dataset size, often at considerable expense in terms of human annotation, data curation, and quality control. The repetition advantage suggests a fundamentally different approach: invest in smaller, higher-quality datasets and train longer.

From a practical standpoint, this enables more efficient development cycles. Teams can iterate on dataset quality with smaller samples, using training token accuracy as a reliable signal for when to stop. The predictability of the saturation point provides clear guidance for resource allocation.

Theoretically, these findings raise important questions about the learning dynamics of large language models. Why does memorization coincide with improved generalization in this setting? Classical learning theory typically treats memorization as antithetical to generalization, yet here they appear aligned. The authors suggest this may relate to the structured nature of chain-of-thought reasoning, where memorizing reasoning patterns transfers to novel problems.

The absence of catastrophic forgetting despite extreme repetition also warrants investigation. Standard neural network theory predicts that repeated exposure to the same examples should lead to overfitting and degraded performance on unseen data. The robustness observed here suggests that reasoning tasks may have different learning dynamics than traditional supervised learning problems.

Takeaways

1. Under fixed computational budgets, training for more epochs on smaller datasets outperforms single-epoch training on larger datasets for chain-of-thought reasoning tasks
2. Training token accuracy serves as a reliable stopping criterion—improvements plateau when models achieve full memorization of training data
3. Extensive repetition (128+ epochs) does not cause catastrophic forgetting on reasoning benchmarks
4. The “repetition advantage” represents a new open problem in understanding LLM training dynamics, particularly the relationship between memorization and generalization
5. Practitioners should prioritize dataset quality over quantity for reasoning SFT, potentially reducing data collection costs while improving performance