KOPA-Bench: Synthesising Tool-Calling Data From Live API Execution

Paper: 2609.05395 Authors: Dain Kim, Eungi Cho, Kyumin Kim, Shinyeong Noh, Kyuseong Lim Categories: cs.AI, cs.CL

The Gap

Data-sovereignty regulations are pushing public institutions toward a specific deployment shape: open-source, on-premise LLM agents that chain multiple tool-calls across live government APIs. The constraint is regulatory rather than technical — the data cannot leave — and it removes the option of calling a frontier model through an API.

That creates a gap with two halves. Open-source models consistently underperform in this multi-step setting — and no existing benchmark measures the gap, because the setting itself is unusual: real government APIs, real institutional tasks, on-premise constraints. So institutions face a deployment requirement, a known capability shortfall, and no measurement instrument for either.

   THE DEPLOYMENT SHAPE REGULATION FORCES

   DATA-SOVEREIGNTY REGULATIONS
     -> public institutions must deploy OPEN-SOURCE,
        ON-PREMISE LLM agents
     -> that CHAIN MULTIPLE TOOL-CALLS across LIVE
        GOVERNMENT APIs
        |
        v
   the constraint is REGULATORY, not technical:
     the data cannot leave
     -> calling a frontier model through an API is NOT AN OPTION
        |
        v
   [A GAP WITH TWO HALVES]
     [1] open-source models CONSISTENTLY UNDERPERFORM in this
         multi-step setting
     [2] NO EXISTING BENCHMARK measures the gap, because the
         setting is unusual: real government APIs, real
         institutional tasks, on-premise constraints
        |
        v
   -> institutions face a deployment requirement, a known
      capability shortfall, and NO INSTRUMENT for either

The Increment

One sentence: Before this paper, institutions had a regulatory mandate, an unmeasured capability gap, and no way to close it; after it, a 145-task benchmark and a synthesis recipe driven by live API execution let a 9B model nearly match an untuned 27B.

Core Mechanism

The benchmark comes first because the gap had to be measurable. KOPA-Bench comprises 145 real-world tasks over Korean open public APIs — real tasks against real government endpoints, which is what makes it a measure of the deployment setting rather than a proxy for it.

The synthesis method is the paper’s technical contribution, and its central idea is that verification should come from execution, not from plausibility. EDGE — an Execution-grounded Dynamic Graph for tool-calling data synthesis — works in three steps:

• Build a graph of how each tool’s output can feed another’s input. This is the candidate structure: which calls are compositionally compatible, derived from the APIs’ shapes. Many such links will not work in practice, and at this stage you cannot tell which.
• Keep only the links that succeed when actually called against the live APIs. This is the decisive step. Rather than asking a model whether a chain looks plausible, EDGE executes it and discards the failures. The result is a graph whose edges are empirically verified rather than hypothesised.
• Traverse these verified links to synthesize executable multi-step trajectories. Because the edges are verified, the synthesised trajectories are executable by construction — which is the property that makes them useful as training data.

That ordering is the methodological point: verified edges first, trajectories second. Synthesising trajectories first and filtering them afterwards would waste generation on chains that could never run and would leave the filter as the quality bottleneck; deriving from a verified graph means every trajectory has a working path through it.

Then the result that connects the recipe to the institutional problem. Fine-tuned with GRPO on the resulting dataset, the authors’ 9B model nearly matches the untuned 27B model from the same family, improving substantially not only on KOPA-Bench but also on BFCL.

Two things about that claim matter. The comparison is against a same-family model three times larger, so the gain is attributable to the data rather than to a different architecture or training lineage — which is the right control. And the transfer to BFCL, a separate function-calling benchmark, is evidence the model learned something about tool-calling generally rather than fitting the 145 tasks. A recipe that only improved the benchmark it was built from would not justify the execution cost.

   [1] THE MEASUREMENT: KOPA-BENCH
         145 real-world tasks over Korean open public APIs
         <- real tasks against REAL GOVERNMENT ENDPOINTS
         -> measures the DEPLOYMENT SETTING, not a proxy

   [2] THE RECIPE: EDGE = Execution-grounded Dynamic Graph
       Central idea: VERIFICATION SHOULD COME FROM EXECUTION,
       NOT FROM PLAUSIBILITY

         (a) BUILD a graph of how each tool's output can feed
             another's input
               <- the CANDIDATE structure, from API shapes
               <- many such links will NOT work in practice,
                  and at this stage you cannot tell which

         (b) KEEP ONLY THE LINKS THAT SUCCEED when actually
             CALLED AGAINST THE LIVE APIs
               <- the DECISIVE step
               <- rather than asking a model whether a chain
                  LOOKS plausible, EDGE EXECUTES it
               -> edges are EMPIRICALLY VERIFIED, not hypothesised

         (c) TRAVERSE the verified links to synthesize
             EXECUTABLE multi-step trajectories
               -> executable BY CONSTRUCTION

       ORDERING IS THE METHODOLOGICAL POINT:
         verified edges FIRST, trajectories SECOND
         <- synthesising first and filtering after would waste
            generation on chains that could never run, and leave
            the filter as the quality bottleneck

   [3] THE RESULT
         GRPO fine-tuning on the resulting dataset
         -> a 9B model NEARLY MATCHES the UNTUNED 27B model
            FROM THE SAME FAMILY
         -> substantial improvement NOT ONLY on KOPA-Bench
            BUT ALSO on BFCL

       WHY THAT CLAIM MATTERS
         - compared against a SAME-FAMILY model 3x larger:
           the gain is attributable to the DATA, not to a
           different architecture or training lineage
         - transfer to BFCL (a separate function-calling
           benchmark) shows it learned TOOL-CALLING GENERALLY,
           not the 145 tasks
         <- a recipe that only improved its own benchmark would
            not justify the execution cost

Think of it as learning a city’s transit system by actually riding it, rather than by reading the map. The map tells you which lines appear to connect: two routes share a station name, so the transfer looks possible. But whether the connection works — whether the platforms are actually linked, whether the timing allows it — you learn only by trying. If you first write out a hundred itineraries from the map and then discover most are impossible, you have wasted the effort. Build the verified connections first, by riding them, and every itinerary you compose from those edges is one that can actually be travelled. That is the same argument as EDGE’s, and it also explains the transfer: what the model learned was the structure of working connections, not the plausibility of a particular set of trips.

Key Concepts

• An institutional benchmark rather than a proxy: 145 tasks over real government APIs. Measuring the actual deployment setting is what makes the gap actionable for institutions facing the same mandate.
• Execution-grounded verification: keeping only tool links that succeed when called live. It replaces “does this chain look plausible” with “does this chain run”, which is the difference between a model’s prior and a fact.
• Edges before trajectories: deriving trajectories from a verified graph rather than filtering synthesised ones. It puts verification upstream of generation, where it is cheap, instead of downstream, where it is the bottleneck.
• Same-family size control: comparing a fine-tuned 9B against an untuned 27B of the same family. It isolates the data as the cause of the improvement.
• Transfer as the justification for cost: substantial gains on BFCL, a separate function-calling benchmark. Without it, the execution overhead would only have bought a better score on the benchmark the recipe was designed around.

Framework Shift

Before (regulatory mandate, unmeasured gap):
  on-premise open-source agents must chain calls across
  live government APIs
  -> open models consistently underperform
  -> no benchmark for the setting
  -> no way to tell where the shortfall is or how to close it

After (measured, with an execution-grounded recipe):
  KOPA-Bench: 145 real-world tasks over live public APIs
  EDGE: candidate graph -> keep only links verified by LIVE
        execution -> traverse to synthesise executable trajectories
  GRPO fine-tuning: 9B nearly matches untuned 27B, and
        improves on BFCL too

From a mandated deployment with a known shortfall and no instrument, to a benchmark for the setting plus a recipe whose verification comes from execution, the core shift is that synthetic data quality depends on checking composition against the real system rather than on judging it by appearance.

Expert Assessment

Problem choice: Excellent, and it takes a constraint seriously rather than treating it as a nuisance. Data sovereignty is not a temporary inconvenience; it is producing a class of deployments where frontier API access is unavailable by law, and the field has been slow to build benchmarks for the settings regulation actually creates.

Method maturity: EDGE’s central choice — verify by execution and derive trajectories from the verified graph — is the right way round, and it deserves emphasis because the naive ordering (synthesise, then filter) is the obvious one and is worse for a reason the paper makes clear. The comparison against a same-family 27B is a well-chosen control, and the BFCL transfer is the check that distinguishes a general capability gain from benchmark fitting. The scope limits are that the benchmark is Korean and government-API-specific, so its transferability to other jurisdictions and API families is untested, and the execution cost of building the graph is not quantified despite being the recipe’s main expense.

Experimental integrity: The three-way evidence structure — a new benchmark showing the gap, a recipe, and a fine-tune showing the gap closing — covers the claim from measurement to remedy. Transfer to an external benchmark is the strongest element because it is the check that could have failed: had BFCL shown nothing, the result would have looked like overfitting to 145 tasks. The main gap is reporting: since verification-by-execution is the expensive part, a reader deciding whether to adopt EDGE needs to know the call volume required per usable trajectory.

Writing quality: The abstract separates the benchmark, the recipe and the result cleanly, and it names the ordering of EDGE’s steps in the order they matter. Because the practical decision is whether to spend the execution budget, a short statement of the yield — how many calls or how many candidate links per usable trajectory — would make the recipe adoptable rather than only convincing.

Verdict: strong accept — it measures a deployment setting that regulation is creating, and supplies a data-synthesis recipe whose defining move is verifying composition against the live system rather than judging it by appearance.

Takeaways

• Verify composition against the real system, not against plausibility. Whether two tools chain is a fact about the deployment, and only execution establishes it.
• Build the verified structure before generating from it. Deriving trajectories from a verified graph is cheaper than filtering synthesised ones, because it puts verification upstream.
• Compare against a larger model from the same family. Same-lineage controls isolate the data as the cause rather than confounding it with architecture.
• Confirm the gain transfers to an external benchmark. Without that check, a synthesis recipe may only be fitting the tasks it was designed around.