ToolCUA: Teaching Agents When to Click and When to Call APIs

Paper: 2605.12481 Authors: Xuhao Hu, Xi Zhang, Haiyang Xu, Kyle Qiao, Jingyi Yang, Xuanjing Huang, Jing Shao, Ming Yan, Jieping Ye Categories: cs.AI

The Gap

Computer use agents can now control computers through both low-level GUI actions (click, type) and high-level tool calls (file operations via API). Prior work treats these as separate: GUI-only agents like Claude Computer Use and UFO execute everything through pixel-level interactions, while tool-augmented agents like Cradle add tools but lack principled switching logic. The result? Agents either take 20 GUI steps to copy a file when one API call would suffice, or prematurely jump to tools when GUI context is still needed.

The core problem: no training data exists for interleaved GUI-Tool trajectories. Collecting real tool execution traces is expensive and brittle (environment dependencies, API changes). Existing GUI datasets like Mind2Web contain only click-and-type sequences. Without trajectory-level supervision showing *when to switch between modalities, agents can’t learn optimal orchestration.

Problem: Hybrid action space (GUI + Tools) but no switching logic
   |
   v
Assumption: Optimal paths exist that interleave GUI and tools
   |
   v
Method: Synthesize GUI-Tool trajectories + RL for switching decisions
   |
   v
Evidence: 46.85% accuracy on OSWorld-MCP (66% over baseline)
   |
   v
Conclusion: Trajectory synthesis + staged RL enables GUI-Tool orchestration

The Increment

One sentence: Before ToolCUA, agents either used only GUI or bolted on tools without learning when to switch; after ToolCUA, agents learn to orchestrate GUI and tools through synthetic trajectory training and reinforcement learning that rewards efficient path selection.

Core Mechanism

ToolCUA operates in three stages. First, it generates training data by taking existing GUI-only trajectories and inserting tool calls at appropriate points. The system builds a “grounded tool library” by analyzing GUI action sequences (e.g., a sequence of clicks to copy a file) and creating corresponding API tools with preconditions derived from the GUI context. This produces diverse interleaved trajectories without manual annotation.

Second, it performs “Tool-Bootstrapped GUI RFT” (Reinforced Fine-Tuning). The agent is warmed up with supervised learning on synthetic trajectories, then trained with single-turn RL at critical decision points—moments where the agent must choose between continuing with GUI or switching to a tool. The RL signal comes from comparing outcomes: did the tool call succeed and shorten the path?

Third, ToolCUA undergoes online agentic RL in a high-fidelity environment (OSWorld-MCP) with a custom reward function. The “Tool-Efficient Path Reward” gives positive signal for successful tool use that reduces steps, negative signal for failed tool calls or unnecessary GUI detours, and neutral signal for pure GUI paths that work but are suboptimal. The agent learns through trial and error which switching points actually pay off.

Stage 1: Trajectory Synthesis
   GUI-only data --> [Analyze patterns] --> Grounded tool library
                                        |
                                        v
                         [Insert tool calls] --> GUI-Tool trajectories

Stage 2: Tool-Bootstrapped RFT
   Synthetic data --> [SFT warmup] --> [Single-turn RL at switch points]
                                                    |
                                                    v
                                            Better switching policy

Stage 3: Online Agentic RL
   [Agent acts in environment] --> [Tool-Efficient Path Reward]
            ^                                      |
            |                                      v
            +-------------- [Policy update] <------+

Think of ToolCUA like training a chef who can both cook from scratch (GUI) and use pre-made ingredients (tools). Stage 1 is creating a recipe book by watching chefs cook everything from scratch, then noting “here you could’ve used canned tomatoes instead of peeling 10 tomatoes.” Stage 2 is practicing those recipes with a mentor who says “yes, switch to canned here” or “no, fresh is better for this dish.” Stage 3 is running a real kitchen where you learn through service: if using canned tomatoes gets the dish out faster without complaints, that’s reinforced; if it ruins the flavor, you learn to stick with fresh.

The key insight: you can’t learn optimal switching by only seeing one modality. The synthetic trajectories provide the “what if” counterfactuals (what if I’d used a tool here?), and the RL stages provide the “does it actually work” ground truth.

Key Concepts

• Grounded Tool Library: Instead of hand-coding APIs, ToolCUA automatically generates tools by analyzing GUI trajectories. If a trajectory shows “click file → click copy → click destination → click paste,” the system creates a copy_file(source, dest) tool with preconditions extracted from the GUI state (file must exist, destination must be writable). “Grounded” means the tool’s semantics come from observed GUI behavior, not abstract specifications. This solves the cold-start problem: you don’t need existing tool documentation to teach an agent about tools.

• Tool-Efficient Path Reward: Standard RL rewards task completion, but that doesn’t distinguish between a 3-step tool path and a 20-step GUI path. ToolCUA’s reward function explicitly penalizes unnecessary length and rewards appropriate tool use. Concretely: +1 for task success, +0.5 for each tool call that reduces steps compared to pure GUI, -0.3 for failed tool calls, -0.1 per extra GUI step beyond the tool-optimal path. This creates gradient toward efficiency, not just correctness.

• Single-Turn RL at Switch Points: Full trajectory RL is sample-inefficient because most steps are unambiguous (obviously click the button). ToolCUA identifies “switch points”—states where both GUI continuation and tool invocation are plausible—and applies RL only there. Think of it like teaching chess: you don’t need RL to learn pawn moves, but you do need it to learn when to sacrifice material for position. This focuses learning where it matters and speeds up training.

Framework Shift

Before (GUI-only or tool-bolted):        After (ToolCUA):

Task --> [Agent] --> GUI actions         Task --> [Agent] --> Decision point
              |                                        |
              v                                        +---> GUI action
         Success (slow)                                |
                                                       +---> Tool call
                                                             |
                                                             v
                                                       Success (fast)

Linear execution, one modality            Branching execution, learned switching

From sequential single-modality execution to hierarchical dual-modality orchestration, the core shift is treating modality selection as a learned decision problem rather than a fixed architectural choice.

Expert Assessment

Problem choice: Real gap. Every practitioner building computer use agents hits this: you add tools to speed things up, but the agent doesn’t know when to use them. The paper correctly identifies that the bottleneck isn’t tool availability but switching logic. However, the framing slightly oversells novelty—tool-augmented agents have existed (Toolformer, Gorilla), but applying this to GUI domains is genuinely new.

Method maturity: The trajectory synthesis pipeline is clever and practical. Grounding tools in GUI behavior sidesteps the annotation bottleneck. The staged training (SFT → single-turn RL → online RL) is sound engineering, not a conceptual breakthrough. The Tool-Efficient Path Reward is reasonable but hand-tuned (those +0.5/-0.3 coefficients smell like hyperparameter search). A simpler baseline would be: just train on synthetic trajectories with SFT—does the RL actually matter? The paper doesn’t ablate this cleanly.

Experimental integrity: OSWorld-MCP is a solid benchmark (real desktop tasks, not toy environments). The 66% relative improvement is impressive, but the absolute accuracy (46.85%) reveals the task is still hard. Baselines are fair: they compare against GUI-only and tool-augmented variants. One concern: the synthetic trajectory generation uses heuristics to decide where tools fit—if those heuristics are too conservative or aggressive, they bias the learned policy. The paper doesn’t deeply analyze failure modes or show where the agent still makes wrong switching decisions.

Writing quality: The method section is dense but complete. The related work undersells connections to hierarchical RL and options frameworks, which have studied action abstraction for decades. The results section focuses on aggregate metrics; case studies showing specific switching decisions would strengthen intuition. The “grounded tool library” concept is introduced late—it should be front and center since it’s the key enabler.

Verdict: weak accept — Solid execution on a real problem with meaningful empirical gains, but the conceptual contribution is incremental (applying known RL techniques to a new domain) and the writing doesn’t fully illuminate the method’s inner workings.

Takeaways

Steal the trajectory synthesis pipeline: If you’re building agents with hybrid action spaces (e.g., code generation + terminal execution, or web navigation + API calls), you can repurpose single-modality data by analyzing patterns and inserting higher-level actions. The key is making those insertions “grounded”—derived from observed behavior, not abstract specs.

Focus RL on decision points, not full trajectories: Most agent actions are deterministic given the state. Apply RL only where genuine ambiguity exists (switch points, exploration-exploitation tradeoffs). This is faster and more sample-efficient than end-to-end RL.

Reward efficiency, not just correctness: If your agent can solve tasks in multiple ways, standard success/failure rewards won’t push it toward better solutions. Add explicit efficiency terms (path length, resource usage, latency) to the reward function.

Beware synthetic data bias: The quality of your synthetic trajectories caps your agent’s performance. If your heuristics for inserting tools are wrong, the agent learns wrong switching logic. Validate synthetic data against real usage patterns before scaling up training.