DataOrchestra: Teaching LLMs to Treat Each Data Example Like a Solo Performance

Paper: 2607.24717 Authors: Zhen Huang, Yikun Wang, Shijie Xia, Pengfei Liu Categories: cs.CL, cs.AI

The Gap

Everyone working on pretraining data knows the pipeline: deduplicate, filter by quality score, maybe do some rule-based cleaning. The dominant paradigm — from C4 to RedPajama to DCLM — defines these steps at the corpus or domain level. You pick a strategy, apply it uniformly to millions of examples, and hope the average behavior works out.

The problem is that “the average” is a fiction. A Wikipedia paragraph with a minor formatting glitch needs surgical editing. A malformed forum post needs to be thrown out entirely. A clean ArXiv abstract needs no touch at all. Applying the same pipeline to all three is either wasteful (cleaning what doesn’t need it) or destructive (over-processing what should’ve been left alone, or under-processing what needed more).

Prior per-example methods exist (e.g., selective filtering via quality classifiers), but they’re binary — keep or drop. Nobody has tackled per-example adaptive processing where the choice includes *how to clean, not just whether to keep.

Problem: Same pipeline applied uniformly to all examples
         |
         v
Assumption: Not all data needs the same treatment
         |
         v
Method: Per-example orchestrator decides {drop, keep, clean}
         |                                 |
         |                                 +-> selects operation(s)
         |                                 +-> generates instructions
         |                                 +-> executes via tool models
         v
Evidence: Stable gains across 11 benchmarks (0.5B-7B),
          compute savings from skipping unnecessary ops
         |
         v
Conclusion: Adaptive per-example processing beats uniform pipelines

The Increment

One sentence: Before this paper, data curation was a one-size-fits-all factory line; after, it’s a concierge service where each example gets assessed and routed to exactly the treatment it needs.

Core Mechanism

DataOrchestra has three layers of decision-making that cascade for each data chunk:

Layer 1 — The Triage. An orchestrator model (a trained classifier) looks at a chunk of raw web data and makes a three-way decision: drop it (too low quality to salvage), keep it untouched (already clean enough), or send it for cleaning. This is the bottleneck decision — it determines whether expensive downstream processing is even needed.

Layer 2 — The Prescription. For chunks that need cleaning, the orchestrator selects which downstream operations to apply from a menu: programmatic fixes (URL removal, whitespace normalization), heuristic rewrites, or LLM-based rewriting with various strategies (paraphrase, expand, restructure). Crucially, it can select *multiple operations in sequence, building a custom mini-pipeline per example.

Layer 3 — The Instruction. For each LLM-based rewriting step, the orchestrator generates a concrete, example-specific instruction — not a generic “clean this up” but something like “remove the navigation artifacts in lines 3-7 and rewrite the fragmented sentence in line 12 as a complete thought.” A downstream tool model (e.g., a smaller LLM) executes this instruction.

Raw Data Chunk
      |
      v
+------------------+
|   Orchestrator   |  <-- trained to make routing decisions
+------------------+
      |
      +-----> Drop (low quality, unsalvageable)
      |
      +-----> Keep (already clean)
      |
      +-----> Clean
                |
                v
         +-------------+
         | Select Ops   |  <-- programmatic / heuristic / LLM-rewrite
         +-------------+
                |
                v
         +-------------+
         | Gen Instruct |  <-- per-example, per-op instructions
         +-------------+
                |
                v
         +-------------+
         | Tool Model   |  <-- executes the rewrite
         +-------------+
                |
                v
         Curated Example

The metaphor that makes it click: Think of DataOrchestra as a hospital emergency room triage system. A patient (data chunk) arrives. The triage nurse (orchestrator) doesn’t treat everyone the same — they assess severity. Some patients get sent home (keep — they’re fine). Some are beyond help (drop — too damaged). The rest get routed to specific specialists (operations), each of whom receives a tailored treatment plan (instruction) rather than a generic protocol. The key insight is the same as in medicine: the most expensive resource isn’t the treatment — it’s the *wrong treatment applied to the wrong patient.

Key Concepts

• Per-example orchestration: Imagine you’re a teacher grading 100 essays. The old way: every essay gets the same red-pen treatment — circle errors, add comments, done. The DataOrchestra way: first scan each essay. Some are perfect (return immediately). Some are incomprehensible (return with a note to rewrite). The rest get custom feedback — “fix your thesis statement,” “this paragraph needs a transition,” “your conclusion contradicts your intro.” The orchestrator is the teacher’s judgment call, made automatically and at scale. The concrete example: two web pages might both have quality score 0.6, but one is a coherent blog post with some HTML junk (clean it — remove the junk) while the other is a SEO keyword dump (drop it — nothing to salvage).

• Instruction generation: This is the trickiest part. Instead of having a fixed rewriting prompt like “make this text cleaner,” the orchestrator writes a *specific prescription for each example. It’s the difference between a doctor prescribing “take medicine” versus “take 200mg of X twice daily with food for 7 days.” The instruction is conditioned on the specific problems detected in the specific example. This matters because generic cleaning instructions tend to over-rewrite good content or under-fix bad content.

• Compute-aware processing: Not every chunk is worth spending LLM inference tokens on. DataOrchestra’s triage step is cheap (a small classifier), and it saves expensive LLM calls by skipping examples that don’t need them. This is the “don’t do surgery when a band-aid will do” principle. The paper reports reduced processing compute alongside better quality — a rare win-win.

Framework Shift

Before (mainstream approach):
                                  
  Raw Data -----> Uniform Pipeline -----> Clean Data
                (same for all)           (one quality level)

  [filter] -> [dedup] -> [clean] -> [score] -> [keep/drop]
       applied identically to every example


After (this paper):

  Raw Data -----> Orchestrator -----> Drop
                       |
                       +-------------> Keep (untouched)
                       |
                       +-------------> Custom Pipeline -> Clean Data
                                         op1 -> op2 -> ...
                                         (example-specific sequence)

From uniform corpus-level pipelines to per-example adaptive routing, the core shift is treating data processing as a classification and planning problem rather than a transformation problem.

Expert Assessment

Problem choice: This is a real gap. The field has been stuck in a “design one pipeline, apply it everywhere” mindset for years. DCLM, FineWeb, and others all define quality at the corpus level. The observation that different examples need different treatments is obvious once stated, but nobody has built a principled framework for it. Good timing too — with inference-time compute and tool-use models becoming mainstream, the “orchestrator + tool” pattern is natural.

Method maturity: The orchestrator-as-classifier approach is pragmatic rather than elegant. Training the router requires labeled examples of what “good processing” looks like, which introduces a bootstrapping problem: you need curated data to train the curator. The instruction generation step is clever but adds latency and complexity. I’d like to see an ablation that separates the value of routing from the value of instruction generation — is the per-example routing alone responsible for most of the gain, or do the custom instructions actually matter?

Experimental integrity: The baselines are reasonable — they compare against individual processing methods and show consistent (if modest) gains across 11 benchmarks. The 0.5B to 7B scale range is appropriate for a data processing paper. The math continued pretraining experiment adds breadth. One concern: the “reduced compute” claim needs more scrutiny — they save on downstream ops but spend on the orchestrator itself. The net savings aren’t fully quantified. Also, the gains are described as “stable” rather than “large,” which is honest but means this is an incremental quality-of-life improvement, not a paradigm shift in model capability.

Writing quality: The paper is competent but not exciting. The related work section likely compresses too much nuance. The most interesting part — how the orchestrator learns to route — deserves more depth. What features does it use? How does it fail? What’s the error analysis? The experimental section could benefit from more qualitative examples showing the orchestrator’s decisions in action.

Verdict: weak accept — The problem framing is genuinely useful and the approach is practical. The gains are real but modest. The paper’s lasting contribution may be the conceptual framework (per-example adaptive processing) more than the specific method.

Takeaways

Steal this: The “triage before treatment” pattern transfers immediately to any data pipeline. Instead of designing a single processing chain, build a cheap classifier that routes examples to different sub-pipelines. This works for dataset cleaning, active learning, and even prompt engineering pipelines.

Steal this too: The instruction-generation step — where the system writes its own task specification per example — is a powerful pattern. Any time you find yourself writing one generic prompt to process a heterogeneous batch of items, ask: “What if I wrote a custom prompt for each item instead?” The orchestrator decides *what to say, the tool model does the work.

Skip this if: You’re looking for a breakthrough in model quality. This is a data engineering paper, not a capabilities paper. The gains are real but in the “1-2% across the board” category. If your data pipeline is already reasonably tuned, the marginal effort of implementing DataOrchestra may not be worth it.