OpenThoughts-Agent: Data Recipes for Agentic Models

Paper: 2606.24855 Authors: Negin Raoof, Richard Zhuang, Marianna Nezhurina, Etash Guha, Atula Tejaswi, Ryan Marten, Charlie F. Ruan, Tyler Griggs, Alexander Glenn Shaw, Hritik Bansal Categories: cs.AI

The Gap

Existing open projects like SWE-Smith, SERA, and Nemotron-Terminal each target a single benchmark (e.g., SWE-bench, terminal tasks). They produce models that excel in that narrow environment but fail to generalize to other agentic scenarios — web navigation, API usage, multi-step reasoning, etc. No one had systematically studied how to curate training data for broad agentic competence. This paper fills that void with a fully open data pipeline, controlled ablations, and a released dataset that demonstrably improves generalization.

+----------------------------------------------------+
| Problem: Agentic models do not generalize          |
| across tasks (single-benchmark focus)              |
+---------------------------+------------------------+
                            |
                            v
+----------------------------------------------------+
| Assumption: A curated, diverse data pipeline       |
| can overcome this limitation                       |
+---------------------------+------------------------+
                            |
                            v
+----------------------------------------------------+
| Method: 100+ controlled ablation experiments       |
| to design pipeline stages (task selection,         |
| generation, filtering). Produce 100K dataset.      |
| Fine-tune Qwen3-32B.                               |
+---------------------------+------------------------+
                            |
                            v
+----------------------------------------------------+
| Evidence: 44.8% avg across 7 agentic benchmarks;   |
| +3.9% over best open model (Nemotron-Terminal-32B) |
| Strong scaling with dataset size.                  |
+---------------------------+------------------------+
                            |
                            v
+----------------------------------------------------+
| Conclusion: Open data curation pipeline works;     |
| released pipeline, dataset, and model.             |
+----------------------------------------------------+

The Increment

One sentence: Before this paper, open agentic models were trained on single-benchmark data and lacked generalization; after, we have a systematic, open data curation pipeline that yields a model outperforming the previous best by 3.9% across 7 benchmarks, with strong scaling properties and full open release.

Core Mechanism

The OpenThoughts-Agent project consists of three major stages: task source curation, data generation, and data filtering / selection. First, they gather tasks from diverse sources (web environments, code execution, multi-turn dialogues) rather than relying on a single benchmark. Second, they use teacher models to generate trajectory data (chain-of-thought plus actions) for each task. Third, they apply filtering criteria (e.g., correctness, diversity, task difficulty) to select a final training set.

The key innovation is the systematic ablation: over 100 controlled experiments vary which task sources are used, how trajectories are generated, and which filtering rules are applied. The authors identify that task diversity is the single most important factor — using tasks from multiple domains yields significantly better generalization than using any single domain, even with less data per domain. The resulting recipe (the “data recipe”) is fixed into a pipeline that produces a 100K example dataset.

+------------------------+     +------------------------+     +----------------------+
| Task Source Curation   | --> | Data Generation        | --> | Data Filtering       |
| (multiple domains:     |     | (teacher model +       |     | (correctness,        |
|  SWE, web, terminal,   |     |  trajectory sampling)  |     |  diversity,          |
|  multi-turn, ...)      |     |                        |     |  difficulty)         |
+------------------------+     +------------------------+     +----------+-----------+
                                                                         |
                                                                         v
+-------------------------------------------------------------------+
| Final Dataset: ~100K examples                                      |
| (training Qwen3-32B)                                               |
+-------------------------------------------------------------------+

Structural metaphor: Think of OT-Agent as a master chef developing a new multi-course menu. The task sources are the ingredient markets: you can’t make a diverse menu if you only buy potatoes (single benchmark). The data generation is the cooking process — each recipe must be tested. The ablation experiments are tasting trials where the chef systematically changes one ingredient at a time (e.g., swap chicken for tofu, add more spice) to see what affects the final dish. The filtering is plating: you discard burnt or bland portions. The final dataset is the cookbook that reproduces the menu at scale. Just as a good chef knows that “variety of produce” matters more than “perfection of one vegetable”, the authors found that task diversity dominates other factors. Anyone can now follow their recipe and bake a better agentic model.

Key Concepts

• Agentic fine-tuning for generalization: Training a language model to not just answer questions, but to perform sequences of actions (click buttons, call APIs, write code, etc.) across different environments. Unlike standard instruction tuning, agentic data must include state transitions and tool use. The key challenge is that a model trained only on terminal commands won’t know how to navigate a web page. OT-Agent’s approach mixes these environments so the model learns transferable strategies.

• Data ablation: A controlled experiment where you systematically remove or modify one component of the data pipeline and measure the effect on final model performance. For example, they vary whether the dataset includes web tasks, or swap the trajectory generation method. This is like testing fertilizer recipes: you plant seeds in different soil mixes and measure growth. The authors did over 100 such experiments to isolate which pipeline choices actually matter.

• Scaling laws for synthetic data: The property that model performance improves predictably as the training dataset grows, assuming the data is generated from a consistent, high-quality pipeline. OT-Agent shows that their pipeline exhibits clean scaling — each doubling of dataset size yields a predictable gain. This means practitioners can extrapolate how much data they need to reach a target performance, similar to how scaling laws work for language model pretraining.

Framework Shift

Before (mainstream approach):
+-----------------------+       +---------------------+
| Single benchmark      | --->  | Fine-tuned model    |
| (e.g., SWE-bench)     |       | Good only on that   |
|                       |       | benchmark; poor on  |
| + low diversity       |       | others              |
+-----------------------+       +---------------------+

After (this paper):
+---------------------------+       +------------------------------+
| Multiple task sources     | --->  | Data curation pipeline       |
| (web, terminal, API, ...) |       | (selection, generation,      |
| + high diversity          |       |  filtering)                  |
+---------------------------+       +---------------+--------------+
                                                    |
                                                    v
+------------------------------------------+       +---------------------+
| Diverse dataset (100K examples)          | --->  | Fine-tuned model    |
| Balanced across domains + quality filter |       | Good across 7       |
+------------------------------------------+       | benchmarks          |
                                                    +---------------------+

One sentence: From single-benchmark overspecialization to multi-domain curated diversity, the core shift is recognizing that training data recipe (task diversity + pipeline) matters more than brute-force scaling of any single task type.

Expert Assessment

Problem choice: Real gap. The field was stalled at “train a model that can do one thing well” vs. “build an agent that can adapt.” This paper directly targets the bottleneck of data curation, which is where most real-world engineering effort goes. Timely and practical.

Method maturity: More clever engineering than breakthrough insight. The systematic ablation is rigorous and valuable — it’s the kind of work that should be done but rarely is in open research. However, there’s no new loss function, architecture, or training technique. The progress comes from careful data design, not algorithmic novelty. That’s fine, but don’t expect a paradigm shift.

Experimental integrity: Good. Baselines are reasonable (Nemotron-Terminal-32B as strongest open model). They control for compute (same base model, same training setup). The scaling comparison across dataset sizes is clean. One red flag: they don’t report variance across seeds for their 44.8% number — is it stable? Also, all benchmarks are simulated; real-world agentic performance might differ.

Writing quality: Competent but dense. The ablation results section (which is the heart of the paper) is a wall of tables with brief commentary. If the authors had dedicated one or two illustrative ablation figures (e.g., bar charts showing the effect of removing each data source), the paper would be much clearer. As is, the reader has to work to extract the qualitative insights.

Verdict: Weak accept — A solid engineering contribution that fills a real gap and provides open resources. It won’t change how we think about AI, but it will change how we build agents, which matters.

Takeaways

• Use the released pipeline as a starting point: If you need a generalist agent, don’t start from scratch. The OT-Agent pipeline gives you a battle-tested recipe to generate diverse training data. Swap in your own task sources if needed.

• The “diversity trumps quantity” rule: Their ablation shows that mixing multiple task domains dramatically improves generalization even if you reduce per-domain examples. Practitioners should prioritize collecting data from 3-4 different environments rather than obsessing over 100K examples from one environment.

• Scaling estimation is now possible: Because the pipeline exhibits clean scaling, you can estimate how much synthetic data you need to reach a target accuracy. This is useful for budgeting compute and data generation costs.

• Open data curation is as valuable as model weights: The released dataset and experimental logs allow others to reproduce and extend the work. This sets a standard for future agentic model development — release the recipe, not just the loaf.