MM-ToolSandBox: When Smart Models Can't Read — A Benchmark That Exposes the Visual Perception Bottleneck in Tool-Calling Agents

Paper: 2607.11818 Authors: Kaixin Ma, Di Feng, Alexander Metz, Jiarui Lu, Eshan Verma, Afshin Dehghan Categories: cs.CV, cs.AI

The Gap

Existing tool-calling benchmarks (ToolBench, API-Bank, ToolAlpaca) evaluate agents on text-only API selection tasks — you describe a goal in words, the agent picks the right tool and fills in parameters from text. But the real world isn’t like that. When a user shows a photo of a restaurant bill and says “split this equally,” the agent must see the total, read the item prices, decide which splitting strategy to use, and call the payment tool with the right amount. This visual grounding → tool execution pipeline is fundamentally untested.

Prior visual agent benchmarks (like VisualWebArena or OSWorld) do include visual input, but they focus on UI navigation rather than tool calling across diverse APIs. They also lack stateful multi-turn interactions where earlier tool outputs change the environment state for subsequent decisions. The missing piece: a benchmark that forces models to extract precise information from images, map it to correct tool parameters, and do so across realistic multi-turn conversations with goal revisions and error recovery.

+------------------+
| Tool benchmarks  |
| (text-only APIs) |
+--------+---------+
         |
         v
+------------------+
| Visual benchmarks|
| (UI navigation)  |
+--------+---------+
         |
         v
+---------------------------+
| MISSING: Visual + Tool    |
| Calling + Stateful Multi- |
| Turn + Cross-Domain       |
+--------+------------------+
         |
         v
+----------------------------+
| MM-ToolSandBox             |
| 500+ tools, 16 domains    |
| Multi-image, multi-turn    |
| Stateful execution env     |
| Failure mode taxonomy      |
+----------------------------+
         |
         v
+----------------------------+
| FINDING: Planning vs.      |
| Precision crossover        |
| 53% failures = visual      |
| extraction errors          |
+----------------------------+

The Increment

One sentence: Before this paper, we evaluated tool-calling agents on text-based tasks and visual agents on navigation tasks — now we can diagnose *where in the visual-perception-to-tool-execution pipeline a model actually fails, revealing that the bottleneck shifts from planning to visual precision as models scale.

Core Mechanism

MM-ToolSandBox has three interlocking components: (1) an automated scenario generation pipeline, (2) a stateful execution environment, and (3) a failure analysis framework.

The scenario generation pipeline uses what the authors call “information-flow-guided planning.” Instead of randomly combining tools, it traces the path that visual information must take to become a tool parameter. For example, if a scenario requires calling a payment API, the pipeline ensures there’s an image containing a receipt, that the receipt has a legible total amount, and that extracting that amount is the natural information flow from visual input → tool parameter. A multi-stage quality filter then prunes scenarios where the visual grounding is ambiguous or the tool chain is trivially solvable. This produces 258 human-verified nominal scenarios plus 50 variants targeting UI-heavy applications.

The execution environment is stateful: when an agent calls tool A and gets output X, that output modifies the environment state, affecting what tool B can do next. The environment supports multi-image inputs arriving progressively across turns — the agent doesn’t see all images upfront but must handle them as they appear in conversation. It also simulates realistic phenomena: users change their minds mid-conversation, tool calls fail and need correction, and environment state mutates between turns.

The failure analysis classifies each failed scenario into one of three categories: planning failures (wrong tool selected, wrong order of operations), precision failures (correct workflow but incorrect information extracted from images), or other failures. This taxonomy is the paper’s analytical backbone.

Scenario Generation              Execution Environment
+------------------+             +-------------------+
| Information-flow |             | Stateful Engine   |
| guided planning  |----+        | (tracks state     |
+------------------+    |        |  across turns)    |
                        |        +--------+----------+
+------------------+    |                 |
| Multi-stage      |    |        +--------v----------+
| quality filter   |----+------->| Multi-turn agent  |
+------------------+    |        | conversation      |
                        |        | (images arrive    |
+------------------+    |        |  progressively)   |
| Human            |    |        +--------+----------+
| verification     |----+                 |
+------------------+              +--------v----------+
                                  | Failure Analysis  |
                                  | Planning? Precision?
                                  | Visual extraction?
                                  +-------------------+

Think of this as a driving test for AI agents. The scenario generator is like the DMV designing test routes: it doesn’t just draw random roads — it traces the information flow that a real driver would encounter. “You’ll need to read a highway sign, extract the exit number, then take that exit.” The execution environment is the actual road: conditions change, other cars appear, and your previous actions (taking a wrong turn) affect what happens next. The failure analyzer is the driving instructor’s clipboard, noting not just “failed” but *why failed. Did the student take the wrong exit (planning failure) or take the right exit but at the wrong number (precision failure)?

Here’s the key insight the metaphor makes load-bearing: a beginner driver (small model) gets lost because they don’t know which turns to take — they fail at route planning. An experienced driver (large model) navigates the route correctly but gets a speeding ticket because they misread the speed limit sign — they fail at visual precision. The paper discovers a crossover point: below a certain model size, the bottleneck is “what should I do?” Above it, the bottleneck is “what am I looking at?” You can’t fix both problems the same way, just as you can’t teach sign-reading to someone who doesn’t know how to drive, and you can’t fix a speeding ticket by teaching route planning.

Key Concepts

• Planning vs. Precision Failure Taxonomy: This is the paper’s diagnostic lens. When an agent fails a task, the taxonomy asks: did the model choose the wrong tools or wrong sequence (planning), or did it choose the right tools but extract wrong information from the image (precision)? Concretely: if a user shows a photo of two hotels and says “book the cheaper one,” a planning failure would be calling the restaurant reservation API instead. A precision failure would be calling the hotel booking API correctly but booking the expensive one because it misread the price. The paper finds that 53% of failures in capable models are precision failures — the model knows what to do but can’t see well enough to do it. This reframes the problem: for large models, better vision encoders may matter more than better reasoning chains.

• Information-Flow-Guided Scenario Generation: Instead of randomly combining tools and images, the pipeline traces how visual information must flow to become a tool parameter. Think of it like designing a scavenger hunt: you start with the treasure (the tool call with its parameters), then work backwards to place clues (visual information in images) along a path that makes the hunt neither trivially solvable nor impossible. This ensures every scenario has a genuine visual grounding dependency — you *must look at the image to succeed, not just read the text description.

• Stateful Multi-Turn Execution: Unlike single-shot benchmarks where each test is independent, this environment remembers. If an agent books a hotel room in turn 3, that room is no longer available in turn 7. If a tool call fails, the error message appears and the agent must recover. This models real tool-calling, where API calls have side effects and conversation context accumulates. The statefulness is what makes the benchmark realistic — and hard.

Framework Shift

Before (mainstream approach):        After (this paper):

Text prompt --> Model --> Tool       Images (multi) --> Model --> Tool calls
                   |                        |              |
                   v                        v              v
            Single-shot eval         Multi-turn stateful execution
            (did it call the         (did it extract correct info
             right tool?)             from images? Did it handle
                                      goal changes? Did it recover
            No visual input           from errors?)
            No state tracking         
            No failure diagnosis      Failure taxonomy:
                                      Planning vs. Precision

From “did the model pick the right API call?” to “can the model read an image, extract the right number, and plug it into the right API call across a multi-turn conversation where things keep changing?” — the core shift is that tool-calling evaluation moves from text-based selection to visual grounding with diagnostic failure attribution.

Expert Assessment

Problem choice: Real gap, well-identified. The tool-calling literature has been text-heavy, and the visual agent literature has been navigation-heavy. Bridging these with a unified benchmark is timely, especially as multimodal LLMs are being deployed as agents. The “planning-to-precision crossover” insight is genuinely interesting and not something I’ve seen articulated this cleanly before. This sits at the intersection of two growing subfields (tool use + visual grounding) and the combination reveals something neither subfield alone would have found.

Method maturity: The automated scenario generation is pragmatic — fully human-authored benchmarks don’t scale, and fully synthetic ones lack quality. The information-flow-guided planning is a reasonable middle ground, though the multi-stage quality filtering feels somewhat ad hoc (the paper could be more transparent about what gets filtered and why). The stateful execution environment is well-designed but not technically novel — it’s engineering, not a new algorithm. The real contribution is the *combination and the analysis framework, not any single technical component.

Experimental integrity: 12 models ranging from 4B open-weight to frontier proprietary — good coverage. The failure analysis adds real value beyond aggregate scores. However, 258 nominal scenarios is on the small side; the paper doesn’t report confidence intervals or discuss scenario-level variance, which matters when the gap between top models is 5-10 percentage points. The 50 UI-focused variants are interesting but too few to draw strong conclusions about that specific domain. No red flags, but the benchmark’s statistical robustness could be more rigorously established.

Writing quality: The paper reads as a competent benchmark paper but not an inspiring one. The scenario generation pipeline section buries the lede — the information-flow concept deserves more prominence and cleaner exposition. The failure analysis section is the best part but could be sharper: the planning-to-precision crossover figure (if they have one) should be the paper’s centerpiece, not buried in experiments. The related work section is thorough but the framing could be tighter — the paper doesn’t clearly position itself against the closest prior work (VisualWebArena, etc.) at the structural level.

Verdict: Weak accept — the planning-to-precision crossover insight is genuinely valuable and the diagnostic framework fills a real need, but 258 scenarios feels thin for a benchmark paper, and the method is more solid engineering than novel insight.

Takeaways

Three things worth stealing:

1. The failure taxonomy for your own agent debugging: When your multimodal agent fails, classify the failure as planning (wrong tool/sequence) vs. precision (right workflow, wrong visual extraction). This immediately tells you whether to invest in better prompting/reasoning (planning fix) or better vision encoders/fine-tuning on visual extraction (precision fix). This diagnostic framework transfers to any visual agent system, not just tool-calling.

2. Information-flow-guided test design: When building your own evaluation suite, don’t randomly combine inputs and expected outputs. Instead, trace the information path from raw input to final output and design test cases that specifically exercise each link in that chain. This produces more diagnostically useful tests with fewer total scenarios.

3. The scaling crossover insight for model selection: If you’re deploying a tool-calling agent, don’t just pick the biggest model. Below a certain capability threshold, your failures are planning errors (fixable with better prompts, chain-of-thought, or structured output). Above it, your failures are visual precision errors (fixable with better image preprocessing, region-of-interest cropping, or fine-tuning on visual extraction tasks). Knowing where your model sits on this curve tells you where to invest engineering effort.