VAKRA: When the Hard Part Isn't Calling the Tool

Paper: 2608.12282 Authors: Ankita Rajaram Naik, Anupama Murthi, Benjamin Elder, Siyu Huo, Raavi Gupta, Abhinav Jain, Praveen Venkateswaran, Abdulhamid Adebayo, Danish Contractor Categories: cs.AI

The Gap

Agent evaluation has quietly split into two disjoint literatures. On one side, tool-use benchmarks (BFCL-style function-calling suites, API-Bank, ToolBench, NESTFUL, AppWorld, ToolSandbox) ask: given a user request and a set of schemas, does the model emit the right call with the right arguments? On the other side, multi-hop retrieval benchmarks (HotpotQA, MuSiQue, MultiHop-RAG, FRAMES) ask: given a document collection, can the model chain evidence across passages? A third, smaller strand — τ-bench and its descendants — adds policy compliance, but typically in one or two hand-built domains (retail, airline).

The enterprise agent nobody has been benchmarking is the one that needs all three at once: pull a customer record from a CRM API, cross-reference it against a policy document written in prose, discover the request is not permitted, and say so. Each existing benchmark, by construction, hands the agent one modality and removes the ambiguity that comes from having to decide where an answer lives.

There’s a second, more technical gap: grading. Most tool benchmarks compare against a single gold trajectory or gold call string, which punishes an agent for finding a different-but-valid route through the API surface. Most RAG benchmarks compare answer strings or use an LLM judge, which is noisy and blind to whether the agent actually did the work.

VAKRA’s bet is that if you close both gaps — one environment, execution-based grading — the failure profile that emerges won’t be about tool-calling mechanics at all.

PROBLEM: an enterprise agent needs APIs + docs + prose rules at once
   |
   +-- prior line A: tool-call benches .... APIs only  [gold call string]
   +-- prior line B: multi-hop RAG ........ docs only  [string / judge]
   +-- prior line C: policy sims .......... 1-2 domains [narrow]
   |
   v
ASSUMPTION: the bottleneck is language-mediated glue
            (which entity / which source / which rule)
            not the mechanics of emitting a well-formed call
   |
   v
METHOD: one live environment
        8000+ executable APIs across 62 domains
        + document corpus + natural-language tool-use policies
        3 settings of rising difficulty
        graded by RE-EXECUTION so many valid paths count
   |
   v
EVIDENCE: best model 70.4 on single-hop endpoint-style
          50-51 on compositional APIs
          > 50 pct relative drop as hop depth grows
          as low as 2.4 on unanswerable-under-policy
   |
   v
CONCLUSION: errors concentrate at entity disambiguation
            and cross-source grounding
            == more tools and better schemas do not fix this

The Increment

One sentence: Before this paper, you could report that your model was good at calling tools *and good at multi-hop retrieval and nobody could check whether it was good at doing both in the same breath under rules it had to read; after it, there’s a single executable environment where that composite skill is measured and the answer is “much worse than you’d guess.”

Core Mechanism

VAKRA is three things stacked. The base layer is a large executable API surface — over 8,000 endpoints spanning 62 domains — plus a document corpus and a set of tool-use policies expressed in natural language rather than code (“do not issue a refund without checking eligibility first,” “if the account is not found, do not infer it from the name”). Sitting on top is a task suite organized into three settings of increasing difficulty: (S1) single-hop calls that probe API interaction style, notably the difference between narrow endpoint-style APIs and compositional ones; (S2) multi-hop chains where the output of one call is the input to the next; (S3) multi-source tasks that require reading documents and APIs together while obeying the stated policies — including queries that are deliberately unanswerable or forbidden, where the correct behavior is refusal or escalation.

The evaluation layer is the piece worth stealing. Instead of comparing the predicted trajectory to a reference trajectory, VAKRA takes the agent’s predicted tool calls and re-executes them against the live APIs, then compares the resulting observable outcome to the reference outcome. Two different call sequences that reach the same end state both get credit. This dissolves the perennial complaint about tool benchmarks — that they reward memorizing one canonical path — and turns correctness into something closer to an integration test than a string match.

The third choice is methodological: every model runs inside the same fixed ReAct loop (thought, action, observation, repeat). No per-model scaffolding, no bespoke planners, no retry policies tuned per system. The claim is that this isolates model capability from agent architecture. It also means the absolute numbers are a floor, not a ceiling — more on that below.

        [ task generator ]
               |
   +-----------+-------------+
   |           |             |
[ APIs ]   [ doc corpus ] [ policy text ]
 8000+        prose         prose rules
 62 domains
   |           |             |
   +-----------+-------------+
               |
        [ query + setting ]
      S1 interaction styles ..... endpoint vs compositional
      S2 multi-hop over APIs .... depth 2 3 4 ...
      S3 multi-source + policy .. incl. unanswerable
               |
               v
      [ fixed ReAct harness ]
       thought -> action -> observation
               ^        |
               +--loop--+
               |
               v
      [ predicted tool calls ]
               |
               v
   [ RE-EXECUTE against live APIs ]
               |
        compare observable outcome vs reference
               |
        +------+-------+
        |              |
     [ match ]     [ mismatch ] -> trace triage:
      credit                       . wrong entity picked
      any valid path                . wrong source consulted
                                    . policy ignored
                                    . malformed call  (rare)

Think of it as the first week of a new hire at a large company, and every part maps.

The 8,000 APIs across 62 domains are the company’s internal systems — HR portal, inventory, billing, ticketing — far more than any one person has seen. Endpoint-style APIs are the single-purpose forms where one button does one thing; compositional APIs are the multi-tab workflows where you must assemble the request yourself. The document corpus is the shared drive and the wiki. The policy text is the employee handbook: prose, not a config file, and it sometimes forbids the obvious shortcut. Multi-hop tasks are the ones where you can’t look up an employee’s manager until you’ve first found their employee ID.

The unanswerable queries are the trap requests every new hire gets — a manager asks for a number that simply isn’t in any system, and the right answer is “I can’t get that,” not a confidently fabricated figure. And the grading is what a good manager actually does: they don’t read your notes to see whether you followed the documented procedure, they re-run your steps and check whether the end state is right. Any route that lands correctly counts. Finally, the fixed ReAct harness is everyone getting the same laptop with the same software — no bringing your own macros — so the review compares people, not tooling.

The paper’s punchline in this metaphor: the new hires can all fill in forms correctly. They fail because they look up the wrong Jane Smith, or they answer from the wiki when the wiki is stale and the API is authoritative, or they never open the handbook.

Key Concepts

• Endpoint-style vs compositional APIs: Two ways to expose the same functionality. An endpoint-style API gives you getOrderStatus(order_id) — one call, one answer, the server did the thinking. A compositional API gives you queryTable(table, filters, join, projection) — powerful, but *you have to know that order status lives in table ORD_HDR, joins to SHIP_EVT on order_key, and that the status you want is the max timestamp row. Same task, radically different burden on the caller. VAKRA finds the gap is roughly 20 points (70.4 to 50-51) for the best model, which tells you something concrete: if you’re designing an API surface for agents to consume, agent-shaped endpoints buy you accuracy that no prompt engineering will recover.

• Execution-based verification with path equivalence: Suppose the task is “cancel the most recent order for user X.” One agent calls listOrders(user=X), sorts, then cancel(id). Another calls getLatestOrder(user=X) then cancel(id). Trajectory matching marks one wrong; VAKRA re-runs both and observes the same order got cancelled, so both are right. The subtlety is that this only works cleanly when the environment can be reset and outcomes are observable — which constrains how state-changing APIs can be designed and is the part of the paper I’d most want to see spelled out.

• Unanswerability under policy: There are two distinct reasons an agent should decline. *Epistemic: the information doesn’t exist in any available source. Deontic: the information exists but the handbook forbids this route to it — say, retrieving salary data without an approval token. Both require the model to notice an absence, which is much harder than noticing a presence: nothing in the context screams “stop.” A 2.4% score here means models essentially always find something to say. For anyone shipping an agent, this is the number that should keep you up at night, because in production the fabricated answer is the expensive failure mode, not the missing one.

Framework Shift

Before (mainstream approach):        After (this paper):

 [ tool-use bench ]                  [ one environment ]
   APIs only                          APIs + docs + handbook
   one gold call string                       |
       |                                      v
   exact match                        agent plans and calls
       |                                      |
   score A                                    v
                                      RE-RUN those calls
 [ multi-hop RAG bench ]                      |
   docs only                                  v
   one gold answer string             compare END STATE
       |                              any working path counts
   F1 or LLM judge                            |
       |                                      v
   score B                            one score + trace triage
                                      . mechanics errors  (few)
 two leaderboards                     . entity errors     (many)
 no rules to obey                     . grounding errors  (many)
 refusal rarely scored                . policy errors     (many)
                                      refusal is a scored answer

From can the model produce the right call to can the model decide which world the answer lives in and whether it is allowed to go there, the core shift is that correctness moves from the syntax of the action to the semantics of the choice.

Expert Assessment

Problem choice: The gap is real, not manufactured. Anyone who has built an enterprise agent knows the failure mode: structured systems and prose knowledge coexist, and the agent’s job is largely arbitration between them. That said, “unified benchmark that merges subfield X and subfield Y” is a crowded and somewhat safe genre in 2026, and the API-scale claim (8,000 endpoints, 62 domains) is the least interesting axis — breadth is cheap when APIs are generated. The genuinely differentiated contribution is the policy-constrained and unanswerable slice. I’d have led with that.

Method maturity: Mostly careful engineering rather than a new idea. Execution-based grading with path equivalence is inherited from the AppWorld / ToolSandbox lineage; the compositional-vs-endpoint contrast is a nice, sharp experimental design choice that costs nothing and yields the paper’s most actionable finding. The overlooked simpler alternative is scale: 100 realistic APIs from actual enterprise systems would likely support the same conclusions with far more external validity than 8,000 synthetic ones. Synthetic API surfaces have a characteristic flatness — naming is too consistent, schemas too regular, documentation too complete — that makes disambiguation both easier (uniform conventions) and harder (near-duplicate endpoints) than reality in ways that don’t cancel out.

Experimental integrity: The fixed-ReAct decision is defensible and clearly stated, but it does real work on the headline numbers. Models trained hard on native function-calling APIs behave differently when squeezed into a text-based ReAct loop, so the ranking may partly measure ReAct-compatibility rather than agentic capability. The 2.4% on unanswerable queries is the number I trust least: abstention rates are notoriously sensitive to system-prompt phrasing, and a single sentence like “if the required information is unavailable, say so explicitly” can move this metric by tens of points. Without a prompt-sensitivity ablation, that figure is a property of the harness as much as of the models. Two more things I’d want and don’t see in the abstract: a human or expert-agent baseline to establish the ceiling, and a discussion of how re-execution handles non-idempotent, state-mutating calls. The claim that failures are language-mediated rather than mechanical is credible and matches my own experience, but it rests on a trace analysis whose taxonomy and inter-annotator agreement need to be visible.

Writing quality: The abstract is disciplined — concrete numbers, no overclaiming about “solving” anything. The corner-cutting is almost certainly in two places: API provenance (how the 8,000 endpoints were generated and validated, and how realistic they are) and the trace taxonomy. The trace analysis is the paper’s intellectual payload — it’s the difference between “here is another leaderboard” and “here is where to spend your research effort” — and in benchmark papers that section is habitually compressed into half a page of pie charts. Rewriting it with worked failure examples, per-category counts, and honest annotation methodology would lift the whole paper a tier.

Verdict: weak accept — a well-constructed and genuinely needed environment whose most quotable numbers are more harness-dependent than the framing admits, but whose diagnostic finding (failures are semantic, not syntactic) is worth having on the record with evidence behind it.

Takeaways

Concrete things to steal:

1. Grade agents by re-execution, not trajectory match. If you’re building internal evals, stop maintaining gold call sequences. Snapshot the environment, run the agent, re-run its calls, diff the observable end state. Your eval stops rejecting correct-but-different solutions and stops needing an update every time you refactor an endpoint.

2. Design agent-shaped endpoints. The 20-point endpoint-vs-compositional gap is a product decision disguised as a benchmark result. If your agent is hitting a generic query interface or a raw SQL-ish surface, you’re paying a large accuracy tax. Wrapping the top twenty task-level operations as first-class endpoints is cheaper than any model upgrade.

3. Split your failure taxonomy along the mechanics/semantics line. Instrument your traces to distinguish malformed calls, wrong-entity selections, wrong-source selections, and policy violations. The paper’s finding is that the first bucket is nearly empty and the rest are full — which means schema polish and JSON-mode fixes have hit diminishing returns, and your effort belongs in disambiguation (ask the user which Jane Smith) and source arbitration (declare which source is authoritative for which field).

4. Make abstention a first-class, separately scored metric — and prompt for it explicitly. Any eval suite without an unanswerable subset is measuring only half the safety-relevant behavior. Corollary from my own skepticism above: since abstention is so prompt-sensitive, that’s also good news operationally — it’s one of the cheapest fixes available to you, and you should ablate it before believing any published abstention number about your model.

5. Fix your harness when comparing models; vary it when forecasting product quality. Conflating the two is how teams end up choosing a model on ReAct-loop numbers and then shipping it in a native-function-calling pipeline where the ranking inverts.