SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning

Paper: 2606.13673 Authors: Seokju Cho, Ryo Hachiuma, Abhishek Badki, Hang Su, Byung-Kwan Lee, Chan Hee Song, Sifei Liu, Subhashree Radhakrishnan, Seungryong Kim, Yu-Chiang Frank Wang Categories: cs.CV, cs.AI

The Gap

Existing agents for spatial reasoning either use single-pass code execution (commit to a full plan before seeing any intermediate result) or structured tool-call APIs (limited in composing operations freely).
Both designs lack the flexibility to adapt analysis based on partial results, especially for open-ended 3D/4D tasks where the next step depends on what was just observed.

[Problem] --> [Assumption] --> [Method] --> [Evidence] --> [Conclusion]
    |               |              |               |
    + Rigid         + Stepwise     + SpatialClaw  + +11.2 avg
      interfaces      code-as-                     across 20
                     action                        benchmarks

The Increment

One sentence: Before – spatial agents were locked into one-shot or rigid tool calls; after – agents can iteratively inspect, compose, and refine perception results via a stateful code kernel, enabling adaptive spatial reasoning.

Core Mechanism

SpatialClaw maintains a persistent Python kernel that holds the input frames + a library of perception and geometry primitives (e.g., depth estimation, segmentation, ray casting).
A VLM-based agent writes one executable cell per step, and the kernel executes it, producing text/visual output that gets fed back into the agent’s context.
On the next step, the agent sees its own previous output and can decide to extract sub-regions, compute distances, overlay heatmaps, or call any primitive.
This loop continues until the answer is ready.

[VLM] --> writes code --> [Stateful Kernel] --> executes --> [Output + Context]
  ^                                                                 |
  +-----------------------------------------------------------------+
                   (previous output becomes new input)

Structural metaphor: Imagine a chef cooking a complex multi-dish meal.
The VLM is the chef, the Python kernel is the kitchen, and the primitives are appliances (blender, oven, knife).
Old approach: write the entire recipe before entering the kitchen (single-pass code).
SpatialClaw: chef enters the kitchen, tastes midway, then adjusts seasoning or decides to roast instead of boil.
Each step writes one command on the kitchen whiteboard (code cell), the appliances execute, and the chef sees the result and decides next.
The stateful kernel is the kitchen that keeps all half-prepared ingredients and tools ready.

Key Concepts

• Stateful execution: Instead of clearing memory after each call, the kernel keeps all variables, images, and intermediate tensors. This allows the agent to build on previous work: compute a depth map in step 1, then crop a region and check if it’s a chair in step 2.
• Code-as-action: The agent writes executable Python, not API calls. This lets it compose arbitrary operations: loop, condition, linear algebra, even inline visualization. It’s far more expressive than predefined tool slots.
• Visual feedback loop: The agent can generate images (e.g., cross-section, heatmap) and treat them as inputs for later steps. This closes the loop between reasoning and observation, crucial for tasks like “how many objects are behind the box?”

Framework Shift

Before (single-pass code):          After (SpatialClaw):
[Agent writes full script]          [Agent writes cell_1]
       |                                    |
[Executes all at once]               [Kernel executes cell_1]
       |                                    |
[Final answer]                       [Agent sees output]
                                            |
                                     [Writes cell_2 based on output]
                                            |
                                     [Loop until answer]

One sentence: From static-batch reasoning to interactive stepping, the core shift is making perception actions reactive to intermediate results.

Expert Assessment

Problem choice: Real gap – spatial reasoning benchmarks demand adaptive multi-step analysis, which rigid interfaces cannot support.
The paper identifies the right bottleneck (action interface design) and tackles it head-on.

Method maturity: Clever but not brute force – the stateful kernel is a simple architectural change, but the real insight is treating code generation as an interactive process rather than a one-shot program.
One could argue that existing agents could be adapted with a similar loop, but nobody did it cleanly.

Experimental integrity: Solid – uses 20 diverse benchmarks, 6 VLMs from 2 families, no cherry-picking.
The +11.2 point gain is consistent across backbones, which rules out VLM-specific artifacts.
One red flag: the paper doesn’t report inference cost (loops may be slower), but for a training-free method this is acceptable.

Writing quality: Clear, though the method section could benefit from a worked example walking through a full reasoning trace.
The appendix contains more details, but the main body feels slightly dense.

Verdict: strong accept – clean problem definition, simple yet effective solution, convincing evidence.

Takeaways

• Practitioners can steal the stateful code kernel pattern: plug it into any agent that needs multi-step tool use, not just spatial reasoning.
• They can also adopt the visual feedback loop idea: let the agent inspect its own output images to make informed decisions.
• The idea of code-as-action rather than tool-call-API is broadly applicable to robotics, navigation, and scientific analysis.
• The 20-benchmark evaluation suite is a gift: anyone building a spatial agent can test against the same leaderboard.