Demystifying Data Organization for Enhanced LLM Training

Paper: 2605.30334 Authors: Yalun Dai, Yangyu Huang, Tongshen Yang, Yonghan Wang, Xin Zhang, Wenshan Wu, Qihao Zhao, Hao Li, Yuanyuan Gao, Kim-Hui Yap Categories: cs.AI, cs.CL

The Gap

Existing LLM training research has focused heavily on data selection—which samples to include or exclude. Methods like curriculum learning, importance sampling, and quality filtering all answer “what to train on.” But modern LLMs are trained for only 1-2 epochs, meaning each sample appears once or twice. In this regime, data ordering becomes critical: the sequence in which samples appear can dramatically affect convergence speed, stability, and final performance.

Prior work either ignores ordering (random shuffling), applies generic curriculum learning (easy-to-hard), or uses ad-hoc heuristics without systematic investigation. The gap: no principled framework exists for understanding how and why data organization matters when training budgets are tight and samples aren’t repeated.

Problem: LLMs trained 1-2 epochs → each sample seen once
   |
   v
Gap: Data selection studied extensively
     Data ordering remains ad-hoc and underexplored
   |
   v
Assumption: Sample-level scores (from selection methods)
            can guide ordering with minimal overhead
   |
   v
Method: Four design principles + two ordering algorithms
        (STR: Stratified Reordering, SAW: Sawtooth Scheduling)
   |
   v
Evidence: Experiments across scales (125M-7B params)
          Pre-training + SFT stages
          Consistent gains in stability and performance
   |
   v
Conclusion: Data ordering is a first-class design choice
            Reusing selection scores makes it nearly free

The Increment

One sentence: Before this paper, data ordering was an afterthought in LLM training; after, it’s a systematic design space with actionable principles and methods that improve training without extra computation.

Core Mechanism

The method operates in two stages. First, it reuses sample-level scores already computed by data selection methods (quality scores, perplexity, diversity metrics). These scores are free—they’re byproducts of existing pipelines. Second, it applies ordering algorithms that arrange samples according to four principles:

1. Boundary Sharpening: Place high-quality samples at epoch boundaries (start/end) to anchor learning and prevent forgetting.
2. Cyclic Scheduling: Repeat quality patterns within epochs (e.g., high-low-high) to maintain consistent gradient signals.
3. Curriculum Continuity: Avoid abrupt difficulty jumps; transition smoothly from easy to hard samples.
4. Local Diversity: Ensure adjacent batches contain varied samples to prevent overfitting to narrow patterns.

Two algorithms implement these principles:

• STR (Stratified Reordering): Divides data into quality strata, then interleaves them to balance difficulty and diversity within local windows.
• SAW (Sawtooth Scheduling): Creates a sawtooth quality pattern—gradual ascent followed by sharp drops—to combine curriculum learning with periodic resets.

Input: Dataset D with pre-computed scores S
   |
   v
[Stratification Layer]
   Divide D into quality bins: D_high, D_mid, D_low
   |
   v
[Ordering Engine]
   STR: Interleave strata in fixed pattern
        [high, mid, low, high, mid, low, ...]
   SAW: Build sawtooth waves
        [low -> mid -> high | low -> mid -> high | ...]
   |
   v
Output: Ordered sequence D' fed to training loop

Think of it like meal planning for an athlete. You have ingredients (data samples) with nutritional scores (quality metrics). Random shuffling is like eating whatever’s in the fridge—it works, but it’s suboptimal. This paper says: arrange meals strategically. Start the day with protein (high-quality samples at boundaries) to set a strong foundation. Alternate between carbs and protein throughout the day (cyclic scheduling) to maintain energy. Gradually increase intensity (curriculum continuity) without shocking the system. And vary your diet (local diversity) so you don’t get bored or deficient. The ingredients are the same, but the sequence changes outcomes—energy levels, recovery, performance. STR is like a balanced meal plan with fixed ratios. SAW is like periodization in training: build up intensity, then reset with a recovery phase.

Key Concepts

• Boundary Sharpening: Imagine training as a journey where the model’s “memory” fades over time. If you end an epoch on low-quality samples, the model’s final state is polluted—it carries noise into the next epoch. If you start an epoch on junk, the model begins from a weak foundation. Boundary sharpening places high-quality samples at epoch start and end, like bookends that anchor learning. Concretely: if you have 1M samples and the top 10% are high-quality, you’d place 50k at the start, 50k at the end, and distribute the rest in between. This ensures the model always begins and concludes training phases on solid ground.

• Cyclic Scheduling: Training dynamics aren’t monotonic—loss doesn’t decrease smoothly. Models benefit from periodic resets where they revisit easier samples after struggling with hard ones. Cyclic scheduling creates a repeating quality pattern (e.g., high-medium-low-high-medium-low) so the model alternates between consolidation (easy samples) and exploration (hard samples). It’s like interval training: sprint, recover, sprint, recover. Without cycles, a long stretch of hard samples can destabilize training; a long stretch of easy samples wastes compute.

• Local Diversity: If consecutive batches are too similar (e.g., all samples about “cooking recipes”), the model overfits to that narrow distribution and forgets broader patterns. Local diversity ensures that within any sliding window of batches (say, 100 batches), you see a mix of topics, difficulties, and styles. Concretely: if you have samples scored by topic clusters, you’d avoid placing all “cluster A” samples together. Instead, you’d interleave clusters so every local window has variety. This prevents the model from getting stuck in local minima.

Framework Shift

Before (mainstream approach):        After (this paper):

[Data Selection]                     [Data Selection]
      |                                    |
      v                                    v
  Random Shuffle                       Principled Ordering
      |                                /   |   \
      v                               /    |    \
  Training Loop                  STR    SAW   Custom
                                  \      |      /
                                   \     |     /
                                    v    v    v
                                  Training Loop

Focus: WHAT to train on             Focus: WHAT + HOW (sequence)
Ordering: Afterthought              Ordering: First-class design
Cost: Zero                          Cost: Still ~zero (reuse scores)

One sentence: From treating data as an unordered set to treating it as a carefully sequenced curriculum, the core shift is recognizing that when the model sees a sample matters as much as whether it sees it.

Expert Assessment

Problem choice: This is a real gap, not manufactured. The field has obsessed over data selection while ignoring ordering, despite clear evidence (from curriculum learning, active learning) that sequence matters. The timing is right—LLMs are now trained for 1-2 epochs, making ordering critical. The problem sits at the intersection of data efficiency and training dynamics, both hot areas.

Method maturity: The principles are sensible but not groundbreaking—boundary sharpening and curriculum continuity are intuitive extensions of existing ideas. The novelty is in systematization: formalizing guidelines and showing they generalize across scales and stages. STR and SAW are simple, which is good—they’re practical. But I’d like to see more exploration of adaptive ordering (adjusting on-the-fly based on training signals) rather than static schedules.

Experimental integrity: Baselines are fair (random, curriculum learning, reverse curriculum). The experiments span multiple scales (125M to 7B), datasets (pre-training and SFT), and metrics (loss, perplexity, downstream tasks). The results are consistent but not dramatic—gains are 1-3% in most cases, which is meaningful but not transformative. One red flag: the paper doesn’t deeply investigate why certain orderings work. The principles are validated empirically, but the mechanistic understanding is shallow. What’s happening in the loss landscape? How do gradients differ?

Writing quality: The paper is well-structured and clear. The principles are articulated crisply. However, Section 4 (experiments) is dense and could benefit from more visual summaries—tables dominate, and it’s hard to extract the big picture. The related work section is thorough but reads like a literature dump. If I were rewriting, I’d add a “failure modes” section: when does ordering not help? Are there datasets or architectures where it’s neutral or harmful?

Verdict: Weak accept — Solid empirical work with practical value, but limited theoretical depth and incremental novelty. It’s a useful contribution that practitioners will adopt, but it won’t reshape the field.

Takeaways

1. Reuse existing scores: If you’re already computing quality metrics for data selection, use them for ordering too. It’s nearly free and often helps.

2. Anchor with quality: Place your best samples at epoch boundaries. This is the lowest-hanging fruit—easy to implement, consistently beneficial.

3. Avoid monotonic curricula: Don’t just go easy-to-hard. Introduce cycles (high-low-high) to prevent forgetting and maintain gradient diversity.

4. Local diversity matters: When batching data, ensure adjacent batches aren’t too similar. A simple heuristic: shuffle within quality strata rather than globally.

5. Transferable insight: The principles (boundary sharpening, cyclic scheduling) apply beyond LLMs—think continual learning, reinforcement learning, or any setting where data arrives sequentially and can’t be repeated indefinitely.