Scaling Laws for Tool-Use Agents via Controllable Environment Synthesis

Paper: 2610.37648
Authors: Andrew Szot, Samira Abnar, Omar Attia, Abbas Kazerouni, Michael Kirchhof, Cheng-Yu Hsieh, Raviteja Vemulapalli, Alexander Toshev
Categories: cs.AI, cs.CL, cs.LG

The Gap

Large Language Models (LLMs) increasingly act as autonomous agents capable of invoking external APIs, querying databases, and executing multi-turn tool calling across diverse software ecosystems (e.g., ToolBench, Gorilla, SWE-bench).

However, training performant tool-use agents has remained an empirical black art constrained by critical data scaling bottlenecks:

1. Scarcity and Privacy of Real-World Tool Telemetry:
   • Production API execution logs contain proprietary user data, private tokens, and confidential company databases, making them impossible to open-source or scale freely.
2. The Naive Synthetic Data Trap:
   • Modern post-training pipelines rely on synthetic environments generated by prompting LLMs.
   • However, existing generation methods naively scale along arbitrary axes—typically generating hundreds of thousands of varied user query phrasings for a tiny, fixed set of 10 to 20 hardcoded API tools.
3. Absence of Predictable Scaling Laws:
   • While pretraining language models adheres to predictable Chinchilla compute and data scaling laws, the community has had zero mathematical scaling laws for agentic tool use.
   • Teams have had no way to predict whether to invest compute in more tools, more tasks, or richer database states.

   THE SYNTHETIC TOOL-USE SCALING DILEMMA

   Traditional Approach:
   Fixed Tools: [Weather, Calendar, Calculator (N=5)]
          │
          ▼ Explode query diversity: 100,000 ways to ask "What's the weather in Seattle?"
   [Trained Agent] ──> Fails catastrophically on any new API with nested schemas!

   THE UNANSWERED SCALING QUESTIONS:
   - Does 2x tools beat 2x tasks?
   - Does database state entropy matter more than prompt variety?
   - Can agent generalization be predicted mathematically before training?

The Increment

One sentence: By systematically generating 458,752 synthetic tasks and training 849 distinct models across four parameter scales with the SynthAPI pipeline, the authors derive the first mathematical scaling laws governing agentic tool-use generalization.

Core Mechanism

The authors developed SynthAPI, a fully controllable generative pipeline that decomposes synthetic tool-use environment generation along three orthogonal axes:

   SYNTHAPI THREE-AXIS ENVIRONMENT SYNTHESIS & SCALING LAW

          Tool Dimension (T)             Database Dimension (D)
   [Diverse Schemas / Argument Trees]    [Relational Tables / State Entropy]
                  \                              /
                   \                            /
                    ▼                          ▼
              ┌──────────────────────────────────────┐
              │ Controllable Environment Synthesis  │
              └──────────────────────────────────────┘
                                  │
                                  ▼ Task Dimension (K)
                       [Multi-Turn Task Prompts]
                                  │
                                  ▼
   Massive Empirical Matrix: 458,752 Tasks × 849 Models (1B to 14B)
                                  │
                                  ▼
   POWER-LAW FORMULATION OF AGENT GENERALIZATION:
   Error(T, D, K, N) = E_0 + α_T T^(-β_T) + α_D D^(-β_D) + α_K K^(-β_K) + α_N N^(-β_N)

1. Three Orthogonal Generation Axes:
   • Tools ($T$): Parameterizes API diversity, argument complexity, return type heterogeneity, and schema depth.
   • Databases ($D$): Parameterizes the underlying relational database state space, row entity count, and stateful mutations.
   • Tasks ($K$): Parameterizes user intent composition, multi-step dependency depth, and language surface variation.
2. Massive Systematic Training Grid:
   • Spanning 849 distinct training runs across four model parameter sizes (1B, 3B, 7B, 14B), the authors systematically varied $T$, $D$, and $K$ under matched training token budgets.
3. The Empirical Tool-Scaling Formula:
   • Downstream out-of-distribution (OOD) generalization error follows a power law jointly determined by tool diversity $T$, database complexity $D$, task volume $K$, and model parameter size $N$: $\mathcal{E}(T, D, K, N) = \mathcal{E}_\infty + \left(\frac{T_0}{T}\right)^{\gamma_T} + \left(\frac{D_0}{D}\right)^{\gamma_D} + \left(\frac{K_0}{K}\right)^{\gamma_K} + \left(\frac{N_0}{N}\right)^{\gamma_N}$
   • Crucially, the exponents reveal an empirical hierarchy: $\gamma_T > \gamma_D \gg \gamma_K$.

The Evidence

The discovered scaling laws revealed transformative practical principles for data engineering in agent training:

Data Scaling Strategy	OOD Unseen Tools (ToolBench)	Multi-Turn API Generalization	Complex State Recovery
Baseline (Scale Task Prompts 10x)	48.2%	51.4%	42.1%
Scale Model Size Only (3B $\to$ 14B)	54.1%	56.8%	49.3%
SynthAPI Pareto Scaling (Scale Tools + DB)	69.8% (+21.6%)	73.5% (+22.1%)	68.2% (+26.1%)

Key scaling law discoveries:

• Tools Over Tasks: Scaling tool diversity ($T$) yields dramatically higher marginal generalization gains than scaling the number of tasks ($K$). Once an agent sees $\sim 200$ task variations per tool, adding more task rephrasings produces near-zero transfer. In contrast, doubling the number of distinct API schemas improves out-of-distribution generalization by +14.2%.
• Database Diversity Is Crucial: Scaling underlying database states ($D$) forces the model to learn error recovery, empty query handling, and state mutation tracking, boosting multi-turn tool success by +26.1%.
• Data Complexity Multiplier: Increasing argument nesting depth and return schema complexity yields a super-linear improvement in the model’s robustness against real-world malformed API calls.

Key Takeaways

1. The First Chinchilla Law for Agents: Tool-use generalization is not random; it adheres to mathematically predictable power laws across tools, databases, tasks, and model size.
2. Stop Generating More Prompts for the Same APIs: The most common industry mistake is generating 50,000 user prompts for 10 APIs. Compute is 5x more effective when invested in synthesizing novel tool definitions and complex databases.
3. Blueprint for Generalist Tool Intelligence: By controlling synthetic environment diversity according to SynthAPI’s scaling exponents, developers can systematically train sub-10B models that outperform much larger models trained on ad-hoc synthetic corpora.

   传统合成工具数据的伪扩展误区

   传统做法：
   固定工具: [天气查询, 日程添加, 计算器 (仅 N=5 个 API)]
          │
          ▼ 暴力扩展自然语言提示词: 生成 10 万种问法 ("西雅图明天冷吗", "查查西雅图天气"...)
   [训练出的智能体] ──> 面对参数深度嵌套、具有状态变更的新 API 时全面崩溃！

   核心追问：
   - 工具数量翻倍 vs 任务指令翻倍，谁对泛化更重要？
   - 底层数据库的状态熵对智能体规划有多大决定性？
   - 我们能否在训练前就精准预测智能体的泛化准确率？

   SYNTHAPI 三维正交环境合成与扩展架构

          工具维度 (Tools: T)                 数据库维度 (Databases: D)
   [API 接口丰富度 / 树状参数嵌套]         [底层关系型数据表 / 状态转移熵]
                  \                              /
                   \                            /
                    ▼                          ▼
              ┌──────────────────────────────────────┐
              │  SynthAPI 形式化可控环境合成流水线  │
              └──────────────────────────────────────┘
                                  │
                                  ▼ 任务维度 (Tasks: K)
                       [多轮复杂依赖任务指令]
                                  │
                                  ▼
   实证矩阵：458,752 项任务 × 849 个模型全量训练 (1B 到 14B)
                                  │
                                  ▼
   工具智能体泛化误差的数学幂律公式：
   Error(T, D, K, N) = E_0 + α_T T^(-β_T) + α_D D^(-β_D) + α_K K^(-β_K) + α_N N^(-β_N)