pySpatial: Teaching Language Models to Think in 3D Through Code

Paper: 2603.00905 Authors: Zhanpeng Luo, Ce Zhang, Silong Yong, Cunxi Dai, Qianwei Wang, Haoxi Ran, Guanya Shi, Katia Sycara, Yaqi Xie Categories: cs.CV

The Gap

Current multimodal LLMs like GPT-4V and Gemini excel at describing what they see in images — identifying objects, reading text, even understanding complex scenes. But ask them “which object is closer to the camera?” or “what would this scene look like from 30 degrees to the left?” and they stumble. The problem isn’t lack of visual understanding; it’s that these models process images as flat projections, lacking explicit geometric reasoning about 3D space. Prior work has tried fine-tuning MLLMs on 3D datasets or adding specialized 3D encoders, but these approaches require expensive retraining and don’t generalize well beyond their training distribution. The fundamental issue: MLLMs have no structured way to convert 2D observations into explorable 3D representations.

The Increment

Before: MLLMs guess at spatial relationships from 2D cues → After: MLLMs write programs that reconstruct 3D scenes and query them explicitly.

Think of pySpatial as a film production pipeline. The MLLM is the director who doesn’t operate cameras or edit footage directly, but instead writes a shooting script (Python code). This script calls upon specialized crew members: the cinematographer (camera pose estimator) figures out where each shot was taken from, the VFX team (3D reconstruction module) builds a digital set from the footage, and the virtual camera operator (novel view renderer) can generate any angle the director requests. The director composes these operations in sequence — first reconstruct the scene, then estimate distances, then render a new viewpoint — turning a passive viewer into an active scene explorer.

The framework provides MLLMs with a library of spatial tools wrapped as Python functions: reconstruct_3d(), estimate_camera_poses(), render_novel_view(), measure_distance(), etc. Given an image sequence and a question like “Is the red cube behind the blue sphere?”, the model generates a program that might first call reconstruct_3d() to build a point cloud, then get_object_coordinates() to locate both objects, and finally compare their depth values. Crucially, this happens zero-shot — no gradient updates, no fine-tuning on 3D data. The MLLM leverages its existing code generation abilities, just with a new API.

The execution is deterministic: the generated Python code runs against actual computer vision libraries (like COLMAP for structure-from-motion, NeRF for novel view synthesis). This grounds the reasoning in geometric truth rather than learned correlations. If the model’s spatial logic is wrong, the program fails or produces nonsensical output, providing a clear error signal.

Key Concepts

Visual Programming for Spatial Reasoning

Most people think of programming as writing instructions for computers. But here’s a different lens: programming is a way to decompose complex problems into sequences of well-defined operations. When you write result = function_a(function_b(input)), you’re saying “first do B, then do A with B’s output.” This compositional structure is exactly what spatial reasoning needs.

Consider the question: “From the camera’s perspective, what’s to the left of the red chair?” A human solves this by: (1) identifying the chair, (2) understanding where the camera is, (3) projecting a leftward direction from that viewpoint, (4) checking what objects fall in that region. Each step is discrete and depends on the previous one. pySpatial lets the MLLM express this same decomposition in code:

chair_pos = get_object_position("red chair")
camera_pose = estimate_camera_pose(images[0])
left_direction = compute_left_vector(camera_pose)
objects_left = query_objects_in_direction(chair_pos, left_direction)

The power isn’t just that this works — it’s that the MLLM can invent new compositions for novel questions without seeing examples. That’s the zero-shot magic.

Structure from Motion (SfM) as Scene Archaeology

Imagine you have 20 photos of a room taken from different angles, but you don’t know where the photographer stood for each shot. Structure from Motion is like being an archaeologist who reconstructs both the room’s layout and the photographer’s path from the photos alone.

Here’s the intuition: if the same corner of a table appears in two photos, but looks slightly different (different angle, different size), that tells you something about how the camera moved between shots. By finding thousands of these correspondences — matching points across images — SfM algorithms solve a giant geometric puzzle. The output is twofold: a sparse 3D point cloud (the “structure”) and the camera position/orientation for each photo (the “motion”).

In pySpatial, SfM is the foundation. Before the MLLM can reason about “what’s behind what” or “how far apart are these objects,” it needs a 3D map. The reconstruct_3d() function wraps tools like COLMAP, which performs SfM automatically. The MLLM doesn’t need to understand the linear algebra inside SfM — it just needs to know: “This function takes multiple images and gives me a 3D scene I can query.”

Zero-Shot via Tool Composition

“Zero-shot” usually means a model handles tasks it wasn’t explicitly trained on. But there’s a spectrum. Some zero-shot methods rely on the model having seen similar tasks (few-shot prompting, for instance). pySpatial’s zero-shot is more radical: the MLLM has likely never seen spatial reasoning problems during training, yet it solves them by recognizing that its existing skill — writing Python code — can be repurposed.

Think of it like this: you’ve never assembled a specific IKEA bookshelf, but you know how to read instruction manuals and use a screwdriver. When faced with the new bookshelf, you don’t need training — you compose known skills in a new sequence. Similarly, GPT-4 knows how to write function calls, handle return values, and chain operations. pySpatial just provides a new “instruction manual” (the spatial tool API). The model reads the docstrings, understands what each function does, and composes them to solve spatial queries.

The key enabler: modern LLMs have strong in-context learning for code. Show them a few examples of API usage (in the system prompt), and they generalize to unseen function combinations. This is why pySpatial works without fine-tuning — the model’s code generation ability is already mature enough to handle novel tool orchestration.

Expert Assessment

Problem significance: Spatial reasoning is a genuine bottleneck for embodied AI — robots navigating homes, AR systems understanding room layouts, autonomous vehicles interpreting 3D scenes. The affected community is large: robotics, AR/VR, autonomous systems, and even accessibility tech (helping visually impaired users understand spatial environments). The problem is worth solving, though I’d note it’s more acute for robotics than for, say, general chatbot applications.

Method maturity: This is a proof-of-concept with promising real-world validation (the indoor navigation experiments are a nice touch), but it’s not deployment-ready. Several limitations stand out:

• Computational cost: Running SfM and NeRF for every query is expensive. The paper doesn’t report inference times, but I’d estimate minutes per query on consumer hardware. That’s fine for offline analysis, problematic for real-time robotics.
• Failure modes: What happens when 3D reconstruction fails (textureless surfaces, reflective materials)? The paper doesn’t discuss error handling or graceful degradation. In practice, SfM can produce garbage on challenging scenes.
• Generalization boundaries: The experiments use relatively clean, well-lit indoor scenes. How does this perform on outdoor environments with dynamic lighting, or cluttered spaces where object segmentation is ambiguous?
• Tool dependency: The framework is only as good as its underlying tools. If COLMAP fails to reconstruct a scene, the MLLM’s program will fail too. There’s no learned fallback.

The authors acknowledge the zero-shot nature means no task-specific optimization, but they don’t deeply explore when this is a feature vs. a bug. For some applications, fine-tuning on domain-specific 3D data might yield better results.

Experimental rigor: The benchmarks (MindCube, Omni3D-Bench) are appropriate, and the 12.94% improvement over GPT-4.1-mini is substantial. However, I have concerns:

• Baseline fairness: The baselines are vanilla MLLMs without access to 3D tools. A fairer comparison would include methods like 3D-LLM or other tool-augmented approaches. The paper positions itself against “pure MLLM” baselines, which makes the gains look larger than they might be against specialized 3D reasoning methods.
• Dataset representativeness: MindCube and Omni3D-Bench are synthetic or semi-synthetic. The real-world navigation experiments are compelling but limited in scope (one environment type). More diverse real-world evaluation would strengthen the claims.
• Ablation studies: The paper lacks ablations on which tools are most critical. Is it the 3D reconstruction? The novel view synthesis? Understanding the contribution of each component would help practitioners adapt the framework.

One red flag: the paper doesn’t discuss failure cases in detail. Every method has breaking points; not showing them suggests either the experiments were too narrow or the authors are overselling robustness.

Verdict: Weak accept — the core idea of visual programming for spatial reasoning is sound and the zero-shot capability is genuinely impressive, but the evaluation needs more depth and the practical limitations need honest discussion.

Takeaways

Tool composition as a reasoning strategy: The big idea here isn’t specific to 3D vision — it’s that you can extend LLM capabilities by giving them APIs to specialized tools and letting them write programs. This pattern works because modern LLMs are strong code generators. Practitioners can steal this for other domains: instead of fine-tuning a model for scientific computing, give it access to NumPy/SciPy and let it write analysis scripts. Instead of training on medical data, provide APIs to diagnostic tools and let it compose workflows.

Explicit intermediate representations beat end-to-end opacity: pySpatial forces the model to construct an explicit 3D scene representation before reasoning. This is slower than an end-to-end neural approach, but it’s *interpretable and debuggable. When the model gets an answer wrong, you can inspect the generated code and see where the logic failed. For high-stakes applications (medical diagnosis, autonomous driving), this transparency is worth the computational cost. The takeaway: don’t always optimize for end-to-end differentiability; sometimes discrete, inspectable steps are better.

Zero-shot via skill transfer, not data transfer: Most zero-shot learning relies on transferring knowledge from similar tasks. pySpatial shows a different path: transfer *skills (code generation) to new domains (spatial reasoning) by providing the right interface. This suggests a general strategy: identify what your model is already good at, then design domain-specific APIs that let it apply those skills. For instance, if your LLM is good at logical reasoning, wrap theorem provers as function calls; if it’s good at language translation, wrap linguistic databases as queryable APIs.

The limits of tool-use without learning: pySpatial’s zero-shot nature is both strength and weakness. It works out-of-the-box but can’t improve from experience. If you deploy this in a specific environment (say, a warehouse), it won’t get better at handling that warehouse’s quirks over time. The lesson: tool-use frameworks are great for rapid prototyping and broad generalization, but for production systems, you’ll likely want a hybrid approach — start with tool composition, then fine-tune on domain data to handle edge cases.