See, Point, Refine: Teaching AI Agents to Click Precisely Through Trial and Error

Paper: 2604.13019 Authors: Himangi Mittal, Gaurav Mittal, Nelson Daniel Troncoso, Yu Hu Categories: cs.CV

The Gap

Computer Use Agents (CUAs) can navigate web pages and simple UIs, but they struggle in dense coding environments where buttons, tabs, and text fields are packed within pixels of each other. Existing approaches like SeeClick, CogAgent, and ShowUI use single-shot coordinate prediction: the model looks at a screenshot, predicts (x, y) coordinates, and clicks. No second chances. When the first click misses by 5 pixels in a crowded IDE toolbar, the task fails. The core problem: these models lack a feedback loop to detect and correct their own errors.

Problem: Dense IDE interfaces require sub-pixel accuracy
    |
    v
Assumption: Single-shot prediction can't self-correct displacement errors
    |
    v
Method: Multi-turn refinement with visual feedback from previous attempts
    |
    v
Evidence: 15-20% improvement in click precision across GPT-5.4, Claude, Qwen
    |
    v
Conclusion: Iterative visual reasoning > one-shot coordinate prediction

The Increment

One sentence: Before this paper, GUI agents clicked once and hoped for the best; after, they click, observe the result, and adjust until they hit the target.

Core Mechanism

The method replaces single-shot prediction with a three-phase loop. First, the agent receives a task (“click the debug button”) and a screenshot. It predicts initial coordinates and clicks. Second, the system captures a new screenshot showing where the cursor actually landed—maybe it’s 8 pixels too far left. Third, the agent sees both the target description and the visual evidence of its miss, then predicts a correction vector. This loop repeats up to 3-5 turns until the cursor lands within an acceptable error radius (typically 5 pixels).

The architecture uses vision-language models (VLMs) like GPT-5.4 or Claude as the reasoning engine. Each turn, the VLM receives: (1) the original task, (2) the current screenshot with the cursor position marked, (3) a history of previous attempts. The model outputs a coordinate adjustment, not an absolute position. The system applies this delta, clicks, and feeds the result back. The key innovation is treating the cursor’s visible position as a training signal—the model learns to interpret “I’m 10 pixels northwest of the target” from raw pixels.

Turn 1:                Turn 2:                Turn 3:
Screenshot -----> VLM -----> Click (x1,y1)     Screenshot -----> VLM -----> Click (x2,y2)     Screenshot -----> Success
   ^                |                              ^                |                              ^
   |                v                              |                v                              |
   +--- Cursor at (x1,y1) -------------------------+--- Cursor at (x2,y2) -------------------------+
        "Too far left"                                  "Almost there"

Think of it like parallel parking. A single-shot model is someone who eyeballs the space, turns the wheel once, and either nails it or doesn’t. This paper’s approach is the normal human process: you pull in, see you’re too far from the curb, back up and adjust, check again, nudge forward. The steering wheel is the coordinate prediction, the rear-view mirror is the visual feedback, and the parking lines are the target UI element. Each adjustment uses information from the previous attempt—you don’t re-plan from scratch, you correct the error you just observed.

Key Concepts

• Visual Feedback Loop: Instead of treating GUI grounding as a one-time perception problem, the system makes it a closed-loop control problem. After each action, the agent observes the outcome (cursor position) and uses that observation to compute the next action. This is analogous to how a thermostat works: measure temperature, compare to target, adjust heating. The “temperature” here is cursor displacement; the “heating” is coordinate adjustment. Without this loop, the agent is flying blind—it has no way to know if its internal model of the UI matches reality.

• Displacement Error Correction: The model doesn’t just predict “click at (450, 320).” It predicts “move 15 pixels right and 8 pixels down from where you are now.” This relative encoding is crucial because VLMs are better at judging spatial relationships (“the cursor is slightly left of the button”) than absolute pixel coordinates. It’s the difference between saying “the coffee shop is at GPS coordinates 37.7749, -122.4194” versus “walk two blocks north from where you’re standing.” The second is easier to execute with noisy perception.

• Sub-Pixel Accuracy in Dense UIs: IDEs pack dozens of interactive elements into a 1920x1080 screen. A typical VS Code toolbar has buttons that are 24x24 pixels, separated by 2-pixel gaps. If your click lands 10 pixels off-center, you might hit the wrong button entirely. “Sub-pixel accuracy” here means the final cursor position must be within 5 pixels of the target center—roughly the size of a small icon. This isn’t about rendering precision; it’s about the tolerance for error in a high-density interface where every pixel matters.

Framework Shift

Before (single-shot):                After (multi-turn refinement):

  Task + Screenshot                    Task + Screenshot
         |                                     |
         v                                     v
    [VLM Model]                           [VLM Model]
         |                                     |
         v                                     v
   Predict (x,y)                         Predict (x,y)
         |                                     |
         v                                     v
      Click once                            Click
         |                                     |
    Success or fail                           v
    (no recovery)                    Capture new screenshot
                                              |
                                              v
                                         [VLM Model] <--+
                                              |         |
                                              v         |
                                      Predict delta     |
                                              |         |
                                              v         |
                                          Adjust & click|
                                              |         |
                                              +---------+
                                              |
                                              v
                                      Success (or max turns)

[One sentence: From blind one-shot prediction to iterative visual servoing, the core shift is treating GUI grounding as a feedback control problem rather than a perception problem.]

Expert Assessment

Problem choice: Real gap. Dense coding interfaces are where CUAs actually break in practice—anyone who’s tried to automate IDE tasks knows single-shot clicking is a coin flip. The authors didn’t manufacture this; they’re addressing a deployment blocker.

Method maturity: Clever but not novel in robotics—visual servoing has been around for decades. The contribution is showing that VLMs can do this without explicit training on displacement correction. The simplicity is a feature: no new model architecture, just a different prompting strategy. However, they don’t explore why 3-5 turns is the sweet spot, or whether there are diminishing returns.

Experimental integrity: Baselines are fair (SeeClick, CogAgent, ShowUI are the right comparisons). The 15-20% improvement is consistent across models, which suggests it’s not a fluke. But the paper lacks failure analysis—when does multi-turn refinement still fail? Are there UI patterns where it gets stuck in local minima? The benchmarks (precision-cua-bench) are new, so we can’t compare to prior work directly.

Writing quality: The abstract oversells (“fundamentally rely” is too strong). Section 3 (method) is clear, but Section 4 (experiments) buries the key result (Table 2) under too much setup. The related work section name-drops papers without explaining what they tried and why it didn’t work. Rewriting Section 2 to include a concrete failure case from existing methods would make the motivation land harder.

Verdict: weak accept — Solves a real problem with a simple, transferable idea, but the evaluation could be deeper and the writing tighter.

Takeaways

Steal the feedback loop pattern: Any agent that interacts with a UI can benefit from this. Instead of “predict action → execute → done,” do “predict action → execute → observe outcome → adjust.” This applies beyond clicking: form filling (type text, see if it appeared in the right field, correct), scrolling (scroll down, check if target is visible, scroll more), even code editing (insert line, check syntax highlighting, fix indentation).

Relative vs absolute coordinates: When your model struggles with precise positioning, switch from predicting absolute targets to predicting deltas from the current state. This reduces the burden on the model’s spatial reasoning and makes errors more recoverable.

Visual feedback as supervision: You don’t need labeled training data for every possible UI. If you can render the outcome of an action (cursor position, form state, etc.), you can use that as a self-supervised signal. The model learns to interpret its own mistakes by seeing them.