The Bitter Lesson of Tool Calling: When Code Beats JSON

Paper: 2608.06370 Authors: Ishan Patel, Sahil Sen, Elias Lumer, Vamse Kumar Subbiah Categories: cs.CL

The Gap

The “tools as code” idea has been floating around for a while. CodeAct (Wang et al.) argued that letting an LLM emit Python instead of structured actions gives it composition for free. Anthropic’s engineering posts on code execution with MCP made the same argument from a token-economics angle: if you have 50 tools, don’t paste 50 schemas into context, expose them as importable stubs and let the model write a script. Several agent frameworks shipped variants. The claim became folklore fast.

What was missing was boring but essential: a controlled comparison on an established function-calling benchmark, with the same tools, the same models, and the only difference being the invocation surface. Prior work either (a) introduced a new agent scaffold and a new benchmark together, so you cannot tell whether the scaffold or the benchmark did the work, (b) measured token savings rather than task accuracy, or (c) tested one model family, so you cannot tell whether the effect is a property of the paradigm or a property of that model’s post-training. And nobody had checked the two conditions that actually break production agents: wide parallel fan-out (call the same tool 20 times) and context rot (the tool call you need is buried under 100k tokens of prior junk).

This paper does the boring thing. 14 models, BFCL v4, two invocation surfaces, three conditions.

   [ PROBLEM ]
   "tools as code" is folklore, not measured
        |
        v
   [ ASSUMPTION ]
   code-capable models have stronger priors for
   writing Python than for emitting tool-call JSON
        |
        v
   [ METHOD ]
   hold tools + models + benchmark fixed;
   vary ONLY the invocation surface
   .------------------------.    .-----------------------.
   | native JSON tool call  | vs | typed Python stubs +  |
   | (one call per turn)    |    | sandbox exec, 1 turn  |
   '------------------------'    '-----------------------'
        |
        v
   [ EVIDENCE ]
   base:      PTC >= JSON in 11 / 14 models
   fan-out:   PTC >= JSON in 13 / 14 models
   ctx rot:   JSON -2.3% avg, PTC ~flat
   GPT-5.6 family: +10.6% over JSON
        |
        v
   [ CONCLUSION ]
   PTC is a viable default, not a niche trick;
   and the gain SCALES with model generation
   (weak models: no gain or worse)

That last clause is the paper’s actual title. The bitter lesson framing: don’t engineer a better JSON schema protocol, just let the model write code and wait for the next generation.

The Increment

One sentence: Before, “tools as code” was a plausible design pattern backed by anecdote and token-count arguments; after, it’s a measured claim with a known failure mode (weak models) and a known robustness advantage (long context), on a benchmark people already trust.

Core Mechanism

The setup is deliberately thin, which is the point. In the JSON condition, you do what every API does: dump tool schemas into the request, model emits a structured call object, harness executes it, result comes back as a tool message, next turn. Each call costs a full model round trip. Twenty parallel calls means twenty result messages threaded back through the conversation (or one batched turn, if the model supports parallel calls, which is uneven).

In the PTC condition, the harness takes the same BFCL tool definitions and compiles them into typed Python stubs — function signatures with type annotations and docstrings, in a module the model can import. The model receives these stubs as context and is asked to write a script. The script runs in a sandbox where the stub bodies are wired to the real tool implementations. Loops, conditionals, list comprehensions, intermediate variables — all of it is just Python, so chaining tool A’s output into tool B’s input never leaves the sandbox and never round-trips through the model. Crucially, execution and result handling collapse into a single agent turn: the model writes one script, the harness runs it, and only what the script prints or returns comes back.

That single-turn collapse is where the context-rot robustness comes from. In JSON mode, every intermediate result is a message in the transcript, so a 20-call task inflates context by 20 payloads, and the model has to re-read all of it on every subsequent turn. In PTC mode, the intermediate results live in Python variables inside the sandbox and never enter the model’s context at all. The model’s context stays roughly constant regardless of how many calls the script makes.

  JSON tool calling (baseline)
  ============================
   model                        harness
    |  emit {"name": f, ...}      |
    |---------------------------->| exec f
    |<----------------------------| tool_result_1  ..... enters context
    |  emit {"name": g, ...}      |
    |---------------------------->| exec g
    |<----------------------------| tool_result_2  ..... enters context
    |            ...              |
    |  N calls = N round trips = N payloads in context

  Programmatic tool calling (PTC)
  ===============================
   tool defs
      |
      v
   [ stub compiler ]  ->  tools.py :
                            def f(x: int) -> Doc: ...
                            def g(d: Doc) -> str: ...
      |
      v
   model writes ONE script
      |
      |   out = [g(f(i)) for i in ids]
      |   print(summarize(out))
      v
   [ sandbox ]  --binds stubs--> real tool impls
      |             (loops, branches, N calls
      |              all happen in here)
      v
   only stdout / return value  .....  enters context
      |
      v
   model, context still small

The metaphor: JSON tool calling is ordering at a drive-through window; PTC is handing over a shopping list.

At the drive-through, you say one item, they hand it to you, you say the next item, they hand it to you. Every item is a full exchange through the window, and everything you’ve received so far is piled on your passenger seat, in your way, for the rest of the transaction. If you want to decide item 3 based on what item 2 turned out to be, that decision has to happen at the window, out loud, with you holding up the queue.

With a shopping list, you write “get 20 apples, if the red ones are bruised get green, then weigh the total and tell me the number” and hand it to a shopper. The shopper walks the aisles, makes the conditional call about red versus green without asking you, and comes back with one number. Your car stays empty. The stub compiler is the store’s aisle directory — it tells you what’s available and in what units, so your list is writable without you having to see the shelves. The sandbox is the store. The single returned number is stdout.

Now the failure mode is obvious in the metaphor too: a shopping list only works if the shopper can read and follow conditionals. Hand a complex list to a shopper who can’t, and you get worse results than the drive-through, where you were supervising each step. That’s exactly the paper’s finding — weak/older models do worse with PTC, and the advantage tracks model generation.

Key Concepts

• Programmatic tool calling (PTC): Instead of the model producing a data structure that names a function and its arguments, the model produces *code that calls the function. Concretely: rather than emitting \{"name": "get_weather", "arguments": \{"city": "Tokyo"\}\} and waiting, the model writes w = get_weather("Tokyo") inside a larger script that might also loop over ten cities and compare them. The function is real — the harness has generated a Python stub whose body dispatches to the actual tool. The model never learns anything new; it just uses a skill (writing Python) that it has vastly more training data for than it does for any particular JSON tool-call format.

• Context rot: The observation that model accuracy on a task degrades as irrelevant material accumulates in context, even when the relevant material is still present and technically retrievable. It’s not a hard context-limit failure — the needle is in the window, the model just gets worse at finding and using it. This is why the paper’s single-turn collapse matters: it isn’t only cheaper, it keeps the model’s working context from filling with tool outputs that are no longer needed. The measured effect here is modest and asymmetric: JSON drops 2.3% on average, PTC stays roughly flat.

• Parallel fan-out: A task that requires calling the same tool many times over different inputs — check inventory at 30 stores, look up 50 tickers. JSON APIs handle this with “parallel tool calls,” which is really just a batched array of call objects; support and reliability vary by model, and the results all come back as separate context payloads. In code, fan-out is a for loop or a list comprehension, which is arguably the single most common pattern in all Python training data. This is where PTC’s win is widest (13 of 14 models), and that’s not surprising.

• BFCL v4: Berkeley Function-Calling Leaderboard, the standard benchmark for whether models call functions correctly — right function, right arguments, right handling of the multi-turn and no-call-needed cases. Using it matters because it’s a benchmark the community has already argued about and calibrated against, which removes the “new method, new benchmark” escape hatch.

Framework Shift

  Before (mainstream approach):          After (this paper):

   [ all tool schemas ]                  [ aisle directory ]
   inlined in context                    typed stubs, importable
          |                                     |
          v                                     v
   model picks ONE tool                  model writes ONE script
          |                                     |
      +---+---+                                 |
      |       |                            .----+----.
      v       v                            |  sandbox |
   turn 1  turn 2  turn 3 ...              |  loop    |
      |       |       |                    |  branch  |
      v       v       v                    |  chain   |
   result  result  result                  '----+----'
      \       |       /                         |
       \      |      /                          v
        v     v     v                     one printed answer
   [ context grows with              [ context flat regardless
     every intermediate ]              of call count ]

   control lives in the              control lives in the
   ORCHESTRATION LOOP                MODEL-WRITTEN PROGRAM

From protocol design to program synthesis: the core shift is that control flow over tools stops being something the harness orchestrates turn by turn and becomes something the model expresses once, in a language it already knows.

Expert Assessment

Problem choice: Real gap, and an unglamorous one, which is a point in its favor. The field had converged on “tools as code is better” through blog posts and framework releases without a clean controlled comparison on shared ground. Somebody had to run this. Where it sits in the trajectory: this is the consolidation paper that arrives after the idea paper, and those are undersupplied relative to their value. The bitter-lesson framing — that the right move is to let capability scaling absorb the problem rather than engineer better calling protocols — is the most interesting claim in the paper, and it’s an argument about *research direction, not just about a scaffold. It’s also the claim with the least direct evidence behind it (see below).

Method maturity: Honestly, there’s barely a method here, and I mean that neutrally. The contribution is an experimental design plus a stub compiler. That’s fine — evaluation papers are allowed to be thin on machinery — but it does mean the paper lives or dies on experimental care, and the reader should judge it as an eval paper, not a methods paper. The one genuinely non-obvious design choice is collapsing execution and result handling into a single turn, which is what produces the context-rot result. A simpler thing being underexplored: JSON tool calling with a *stateful result store (return handles, not payloads) would capture much of the context benefit without requiring code generation, and would work for non-code models. The paper doesn’t try it, and it’s the obvious middle baseline.

Experimental integrity: The baseline is fair in the important sense — same tools, same benchmark, same models. But I’d want to know how hard the JSON baseline was tuned. Native tool calling is very prompt- and schema-sensitive, and PTC gets a fresh, presumably well-iterated prompt. If the JSON condition is “whatever the provider’s default looks like,” the comparison quietly favors PTC. Second concern: 11-of-14 and 13-of-14 are vote counts, not effect sizes. “Matches or exceeds” bundles ties with wins, and on BFCL, differences of a point or two are within run-to-run noise unless you’re averaging multiple seeds. I’d want per-model deltas with variance before believing the headline framing. Third: the +10.6% on the GPT-5.6 family is doing a lot of rhetorical work for one family, and the “tracks model capability across generations” claim is drawn from a scatter of 14 heterogeneous models across different labs, training recipes, and code-capability levels — that’s suggestive, not a scaling law. Fourth: the context-rot number, 2.3% average degradation, is small. It’s directionally consistent with the mechanism, but it’s not a dramatic result and shouldn’t be sold as one. Finally, PTC requires a sandbox with real execution; any comparison that ignores sandbox failures, timeouts, and syntax errors as a cost category is understating the operational difference.

Writing quality: The abstract is doing the paper’s best work, which is usually a warning sign. The stub compiler deserves a full section — what happens with tools whose arguments are unserializable, with tools that need auth, with tools that return huge payloads the script must summarize? Those are exactly the details a practitioner needs, and they’re where PTC gets hard in production. The section I’d rewrite is the analysis of the three models where PTC *loses. That’s the most informative data in the paper and the place where the bitter-lesson thesis is either earned or not. If the losses are all on weaker or non-code-tuned models, say so with evidence and the thesis stands; if there’s a strong model in there that regresses, the story is more complicated and more interesting. Skipping that analysis is the paper’s biggest missed opportunity.

Verdict: weak accept — a needed, well-scoped controlled comparison whose headline claims are stated more strongly than vote-count metrics and small effect sizes support, but which nonetheless gives the community shared ground on a question everybody had already answered by assumption.

Takeaways

Things you can actually use:

Handles, not payloads. The transferable insight isn’t “write code” — it’s that intermediate results should live somewhere the model can *reference without reading. You can apply this without adopting PTC at all: have your tools return an ID plus a one-line summary, keep the full payload in a side store, and give the model a fetch(id) tool. Most of the context-rot benefit, none of the sandbox.

Fan-out is a loop, so make it one. If your agent’s workload is dominated by same-tool-many-inputs patterns, this is where code invocation wins widest and most reliably. It’s also the easiest thing to migrate — you don’t need to convert your whole tool surface, just the hot fan-out paths.

Gate PTC on model capability, and measure it. The paper’s real operational lesson is that this isn’t a strict upgrade. If you’re routing across a model tier (cheap model for easy turns, frontier model for hard ones), the invocation surface should probably be part of what you route on. Don’t assume the pattern that works on your best model transfers to your cheap one.

Typed stubs as the tool interface. Even if you keep JSON calling, generating typed Python stubs from your tool registry is a useful artifact: it forces you to have real types, it’s readable documentation, and it makes the migration cheap later. Cheap to build, useful regardless.

A framing worth stealing for your own eval work. Hold everything fixed and vary one surface, across a generation range of models, on a benchmark you didn’t create. Then report whether the effect grows with capability. That last step is what separates “this trick works” from “this trick is the future,” and most scaffold papers skip it.

What not to take: the +10.6% number. It’s one family, one benchmark, and the paper doesn’t give you the variance.