VLM-IE3D: Teaching Vision-Language Models to See in 3D with Dual Geometry Tokens

Paper: 2607.21595 Authors: Wenhao Li, Xueying Jiang, Quanhao Qian, Deli Zhao, Ran Xu, Shijian Lu, Gongjie Zhang Categories: cs.CV, cs.AI, cs.LG

The Gap

Vision-language models have made impressive strides, but there’s a stubborn blind spot: they’re fundamentally flat-thinkers. Models like LLaVA, Flamingo, and their descendants process 2D image patches and text tokens beautifully, but ask them “what’s behind the couch?” or “how far is the cup from the edge?” and they stumble. The core issue is architectural—these models never learned to reason about depth, occlusion, or spatial relationships in 3D.

Some prior work tried to fix this by piping in explicit 3D data: depth maps from sensors, point clouds from LiDAR, or pre-reconstructed meshes. This works in controlled settings but breaks the elegant “just give it a photo” paradigm and limits deployment to scenarios with specialized hardware. Others attempted to learn 3D implicitly through video, but these approaches typically produce a single monolithic representation that doesn’t distinguish between high-level spatial priors (“this scene has a table in the center”) and fine-grained geometric details (“the table’s left edge is at coordinates X, Y, Z”).

The logical path from this gap to the paper’s contribution:

Problem: VLMs are 2D-only, struggle with 3D reasoning
    |
    v
Assumption: Dual geometry (implicit + explicit) > either alone
    |
    v
Method: IGTs (high-level) + EGTs (detailed) fused via adapter
    |
    v
Evidence: Consistent gains across 4 different 3D tasks
    |
    v
Conclusion: RGB-only VLMs can achieve strong 3D awareness

The Increment

One sentence: Before this paper, VLMs either ignored 3D entirely or required expensive 3D inputs; after, they can learn rich 3D spatial understanding from ordinary RGB video alone by decomposing geometry into implicit and explicit pathways.

Core Mechanism

VLM-IE3D works by decomposing 3D understanding into two complementary streams that feed into a standard VLM backbone. The key insight is that “3D knowledge” isn’t monolithic—some of it is intuitive and holistic (knowing a scene has depth without measuring it), while some is precise and structural (knowing exact surface positions). The framework captures both.

The first stream produces Implicit Geometry Tokens (IGTs). These are learned from video frames using a geometry encoder that extracts high-level spatial priors—things like “this region is probably closer than that one” or “there’s likely a wall behind that object.” These tokens are compact, abstract, and capture the gestalt of the 3D scene. They don’t tell you exact coordinates, but they encode the kind of spatial intuition that lets you navigate a room in dim light.

The second stream produces Explicit Geometry Tokens (EGTs). This pathway actually reconstructs 3D attributes—depth, surface normals, or volumetric structure—from the video, then encodes these into tokens that carry precise geometric information. Think of these as the detailed architectural blueprints that complement the intuitive spatial awareness.

A 3D-aware adapter then fuses these two geometry streams with the standard 2D visual features from the VLM’s vision encoder. This adapter uses cross-attention mechanisms to let the model selectively attend to whichever type of geometric information is most relevant for the current task.

Video Frames
    |
    +--[Geometry Encoder]--> IGTs (implicit, holistic)
    |                              |
    |                              |
    +--[3D Reconstruction]--> EGTs (explicit, detailed)
    |                              |
    +--[Vision Encoder]---> 2D Visual Tokens
    |                              |
    +------------------------------+
    |
    v
[3D-Aware Adapter: cross-attention fusion]
    |
    v
[Fused Tokens] --> [LLM Backbone] --> Task Output

Here’s a structural metaphor that makes this click: imagine you’re a real estate agent showing a house to a client who can’t visit in person. You have two tools. First, you walk through the house yourself and develop an intuitive feel—“the living room feels spacious, the hallway is narrow, the kitchen opens up nicely.” That’s your IGT layer: embodied, holistic, captures the vibe of the space. Second, you have detailed floor plans and measurements—“the living room is 15x20 feet, the counter is 36 inches high.” That’s your EGT layer: precise, structural, exact coordinates. Now, when your client asks questions, you draw on both. “Will my couch fit?” requires the floor plan (EGT). “Does the room feel cozy or cavernous?” requires the lived experience (IGT). The 3D-aware adapter is your brain switching between these two knowledge sources depending on what’s being asked. The VLM backbone is the conversation—you’re synthesizing all this spatial knowledge into natural language responses. Without the floor plans, you’d be guessing at dimensions. Without the lived experience, you’d miss the qualitative feel. VLM-IE3D argues that VLMs need both.

Key Concepts

• Implicit Geometry Tokens (IGTs): Think of these as the “spatial intuition” encoded in token form. When you walk into a room, you instantly know the ceiling is above you, the floor is below, and that table is probably a few feet away—even without pulling out a measuring tape. IGTs capture this kind of high-level geometric prior. They’re learned from video by a geometry encoder that sees multiple frames and extracts patterns like relative depth, occlusion relationships, and scene layout. The “implicit” part means they don’t give you exact 3D coordinates; they encode spatial relationships as learned representations. It’s the difference between saying “the vase is on a surface that’s roughly waist-height in front of me” versus “the vase is at coordinates (2.3, 0.8, 1.1).”

• Explicit Geometry Tokens (EGTs): These are the opposite approach—actual 3D reconstruction squeezed into tokens. The framework takes video frames, runs them through a 3D reconstruction module to produce things like depth maps or surface geometry, then encodes these into compact token representations. The “explicit” part means these tokens carry concrete geometric data: how far away things are, what angles surfaces meet at, where boundaries between objects lie. A concrete example: if IGTs tell you “there’s a tall object to the left,” EGTs tell you “there’s a 1.8-meter cylinder at position (x, y) with a 0.3-meter radius.” The key contribution is showing that having both types—not just one—significantly improves 3D reasoning.

• 3D-Aware Adapter: This is the plumbing that makes the dual-geometry approach work. In standard VLMs, visual tokens go straight to the language model. Here, the adapter sits between the vision encoder and the LLM, using cross-attention to mix in geometric information from both IGTs and EGTs. “3D-aware” means the adapter is specifically designed to handle spatial information—it’s not just concatenating tokens, but learning which geometric cues matter for which parts of the visual input. The adapter learns, for example, that when the text mentions “behind,” it should attend more to depth-related tokens, or when it mentions “shape,” it should weight explicit geometry tokens more heavily.

Framework Shift

Before (mainstream approach):              After (this paper):

[Image/Video]                              [Video Frames]
    |                                          |
    v                                          v
[Vision Encoder]                         [Vision Encoder]
    |                                      |       |
    v                                      v       v
[2D Visual Tokens]                   [IGTs] [EGTs] [2D Tokens]
    |                                      \   |   /
    v                                       \  |  /
[LLM]                                    [3D-Aware Adapter]
    |                                          |
    v                                          v
[Text Output]                            [Fused Representation]
                                               |
                                               v
                                          [LLM Backbone]
                                               |
                                               v
                                          [3D-Aware Output]

From single-stream 2D processing to dual-pathway 3D geometry injection, the core shift is decomposing spatial understanding into complementary implicit and explicit channels rather than hoping a flat vision encoder will somehow learn 3D from 2D pixels.

Expert Assessment

Problem choice: This is a genuine gap. The 2D limitation of VLMs is a real bottleneck for robotics, embodied AI, and AR applications. The paper positions itself clearly against two prior camps (3D-input-dependent methods and 2D-only methods) and argues for a middle path. It’s not the most novel problem formulation—people have been talking about 3D-aware VLMs for a while—but the specific angle of dual implicit/explicit decomposition is well-motivated.

Method maturity: The architecture is more clever than brute-force. Decomposing 3D geometry into implicit and explicit pathways is a principled design choice that echoes similar successful decompositions in other fields (think of how motion capture uses both marker positions and learned pose priors). That said, the 3D reconstruction module for EGTs is borrowed from existing work, and the implicit pathway’s geometry encoder likely shares DNA with video representation learning methods. The contribution is more in the integration and the adapter design than in any single component being radically new.

Experimental integrity: The experiments cover four distinct 3D tasks (video detection, visual grounding, dense captioning, spatial reasoning), which is good breadth. The baselines include relevant comparisons. However, I’d want to see ablation studies that rigorously separate the contribution of IGTs vs. EGTs—does one pathway dominate, or are they truly complementary? Also, the paper claims “RGB-only” design, but the 3D reconstruction for EGTs presumably requires training with some form of 3D supervision at some point—this should be clarified more explicitly.

Writing quality: The abstract is dense but well-structured. The main weakness is likely in the method section—explaining how the geometry encoder and 3D reconstruction module work in enough detail for reproducibility while keeping the narrative flowing. The 3D-aware adapter description could also benefit from clearer diagrams showing the attention patterns. If I were rewriting one section, it would be the related work: a more systematic taxonomy of existing approaches would strengthen the positioning.

Verdict: weak accept — The dual implicit/explicit geometry decomposition is a clean, well-motivated idea with solid empirical support across multiple tasks. It’s incremental rather than revolutionary, but it’s the right kind of increment.

Takeaways

The most transferable idea here is the dual-representation decomposition pattern: when you need a model to handle two aspects of the same phenomenon (here, holistic vs. precise geometry), don’t force one representation to do everything—build parallel pathways and learn to fuse them adaptively. This pattern shows up in multi-scale vision, in audio processing (mel spectrograms vs. raw waveforms), and in NLP (syntactic vs. semantic parsing). The 3D-aware adapter’s cross-attention fusion mechanism is directly applicable whenever you need to combine heterogeneous feature streams into a backbone that expects homogeneous input.

Another steal-worthy idea: the “from video alone” pipeline. If your downstream task needs 3D understanding but you only have RGB video, pre-training a geometry encoder on video frames (learning depth, occlusion, spatial layout implicitly) can inject 3D inductive biases without requiring any 3D sensor data. This is practical for scaling to large datasets where LiDAR or depth cameras aren’t available.