SceneBind: Teaching Multimodal Models Where Things Are, Not Just What They Are

Paper: 2607.15265 Authors: Mingfei Chen, Zijun Cui, Ruoke Zhang, Hyeonggon Ryu, Eli Shlizerman Categories: cs.CV, cs.AI, cs.MM, cs.SD

The Gap

Omni-modal encoders like ImageBind and CLIP-based architectures have gotten remarkably good at matching what is in a scene across modalities — you hear a dog barking and the model connects it to images containing dogs, or you read “barking dog” and it finds the right frame. But they represent scenes as single monolithic vectors, which means they collapse all spatial structure. You can retrieve a scene that contains a dog, but you can’t ask “where is the dog relative to the camera?” or ground an audio event to a specific region in the image. Prior attempts at spatial grounding either require task-specific heads trained from scratch, or they bolt on localization as a separate post-hoc step disconnected from the semantic representation.

The gap is precise: there’s no unified representation that simultaneously encodes what each object is (semantics) and where it sits in 3D space (spatial attributes) within a single omni-modal embedding — and does so in a way that’s compatible with the large pretrained encoders everyone already uses.

Existing omni-modal encoders:
  [Scene] > (Single Encoder) > [Global Vector] > (Cosine Sim) > Retrieval
                                         |
                                  Good at WHAT
                                  No WHERE
                                  No object-level structure

Gap:
  Cannot answer "WHERE is the dog?"
  Cannot ground audio to image regions
  Cannot align objects across modalities

SceneBind path:
  [Scene] > (Pretrained Encoder + Spatial Slots) > [Global + Object Slots]
       |
       +--- Global: scene-level semantics (WHAT)
       +--- Slots: per-object semantics + 3D position + uncertainty (WHERE)
            |
            v
       (SceneBind Matching: global sim + object alignment)
            |
            v
       Scene retrieval + Object grounding + Zero-shot transfer

Conclusion:
  Adding explicit spatial slots on top of pretrained semantics
  is complementary, lightweight, and unlocks grounding tasks.

The Increment

One sentence: Before this paper, omni-modal scene encoders could tell you *what is in a room but not where; after this paper, a single representation captures both — with only a handful of extra tokens and no retraining of the backbone.

Core Mechanism

SceneBind’s architecture has two layers built on top of any frozen pretrained semantic encoder (e.g., ImageBind, CLAP). The first layer is a global semantic embedding — a scene-level vector that captures the overall meaning, just like what you’d get from existing encoders. The second layer is a set of semantic-spatial slots: a fixed number of learnable tokens, each responsible for attending to one object or region in the scene. Each slot extracts two things simultaneously: a semantic identity (what the object is) and a spatial attribute (where it is in 3D, including azimuth, elevation, distance, and an uncertainty estimate).

The data flow works like this. Given a scene from any modality (an image, a binaural audio clip, or a text description), the pretrained encoder produces per-patch or per-frame features. These features are fed into the slot attention module, where the learnable slots compete to attend to different objects. Each slot outputs a semantic-spatial embedding. The full scene representation is then the concatenation (or pooling) of the global embedding and all slot embeddings. Training uses a contrastive objective that aligns both the global vectors and the slot-level correspondences across modalities, with a special matching loss that enforces object-level alignment.

Input (any modality)
    |
    v
[Frozen Pretrained Encoder] --> patch/frame features
    |                              |
    v                              v
[Global Pooling]          [Slot Attention Module]
    |                         |    |    |
    v                         v    v    v
[Global Embedding]        [Slot1 Slot2 ... SlotN]
    |                      |     |         |
    |                      v     v         v
    |                   (sem + spatial + uncertainty) per slot
    |                         |
    +----- Concatenate ------+
    |
    v
[Unified Scene Representation]
    |
    v
[SceneBind Matching]
    |-- Global cosine similarity (scene retrieval)
    +-- Object-level alignment (cross-modal grounding)

Think of it like a city directory. The global embedding is the city’s overall character — “industrial port town” vs “romantic European capital.” That gets you to the right city. But when someone says “meet me at the café by the river,” you need the directory’s individual entries: each landmark has a *name and type (semantic — café, park, bridge) and an address on the map (spatial — coordinates, distance from the river, facing direction). SceneBind’s slots are exactly these directory entries. When you query across modalities — you hear street noise and want to find the matching visual scene — SceneBind doesn’t just check “is this the same city?” (global matching), it also checks “do the landmarks line up? Is the café where it should be relative to the river?” (slot-level matching). The uncertainty estimate is like noting “the café is probably on the north bank, but I’m not fully sure.” Without the directory, you’d only know you’re in the right city but have no way to find the café.

Key Concepts

• Semantic-Spatial Slots: Imagine you’re at a party and you close your eyes. You can hear different people talking from different directions. Each voice tells you two things: *who they are (semantic — your friend Alex, a stranger) and where they are (spatial — behind you to the left, three meters away). Semantic-spatial slots work the same way. Each slot is a learnable “listener” that latches onto one object or sound source in a scene, extracting both what it is and where it is. The key trick is that slots compete for attention (via slot attention), so they naturally specialize — one slot grabs the dog, another grabs the car, a third grabs the background hum. Crucially, these slots are modality-agnostic: the same slot that represents “dog, 2 meters left” in an image can represent “barking, 2 meters left” in audio.

• Uncertainty in Spatial Attributes: Not every signal gives you precise spatial information. If someone says “there’s a dog somewhere in the park,” you know the semantics but the spatial location is vague. SceneBind handles this by predicting not just a spatial coordinate but also an uncertainty distribution for each slot. This means the model can express “I’m very confident the guitar is at azimuth 45 degrees” (tight uncertainty) vs “there’s music playing but I’m not sure exactly where” (wide uncertainty). This is modeled as a learned variance alongside the position estimate, and it prevents the model from being overconfident when the signal is ambiguous.

• SceneBind Matching: Standard multimodal retrieval just compares two global vectors with cosine similarity. SceneBind Matching is more nuanced. It computes a global similarity score (are these the same scene overall?) AND an object-level alignment score (do the objects match up in type and position?). The final score is a combination of both. This matters because two scenes can have the same global “vibe” but completely different object layouts — SceneBind Matching can tell them apart. Conversely, two scenes with different lighting or camera angles but the same object arrangement will score high on object alignment even if global similarity is moderate.

Framework Shift

Before (mainstream):                  After (SceneBind):
                                     
[Image/Audio/Text]                   [Image/Audio/Text]
      |                                    |
      v                                    v
[Pretrained Encoder]                 [Pretrained Encoder]
      |                                    |
      v                                    v
[Global Vector]                      [Global Vector] + [Spatial Slots]
  (one per scene)                      (one global + N per object)
      |                                    |
      v                                    |
[Cosine Similarity]                  [SceneBind Matching]
      |                                    |-- global sim
      v                                    +-- object alignment
[Scene Retrieval]                    [Scene Retrieval + Object Grounding]
  (what only)                          (what + where)

From monolithic scene vectors to structured semantic-spatial decompositions, the core shift is that scenes are no longer opaque blobs — they have internal structure that you can query, align, and ground.

Expert Assessment

Problem choice: This is a genuine gap. The field has been coasting on global-vector representations for multimodal retrieval for a while, and everyone knows spatial understanding is the next frontier. The paper is well-positioned at the intersection of representation learning, 3D vision, and audio-visual understanding. It’s not a manufactured problem — the inability to ground audio events to image regions or do object-level cross-modal matching is a real bottleneck.

Method maturity: The approach is more clever engineering than brute force. Slot attention is a well-understood mechanism (borrowed from object-centric learning), and the key insight — that you can add spatial modeling with just a few extra tokens on top of frozen pretrained encoders — is elegant. The risk is that slot attention can be brittle: it assumes a fixed number of objects, and in complex scenes with many overlapping sources, slots may not cleanly decompose. The paper doesn’t deeply analyze failure modes here. There’s also a question of whether simpler approaches (e.g., explicit 3D detection heads followed by feature pooling) would work just as well with more engineering effort.

Experimental integrity: The baselines include reasonable comparisons (ImageBind, AudioCLIP, and other omni-modal methods). The creation of a new binaural audio-visual dataset with spatial annotations is a genuine contribution, but it also raises a flag — when you create your own benchmark, you have an inherent advantage in knowing its distribution. I’d want to see results on external spatial grounding benchmarks to be fully convinced. The zero-shot transfer results on audio-visual localization are encouraging and suggest the representation is learning something real, not just overfitting to their annotations.

Writing quality: The paper is clearly written and the figures are helpful. The main weakness is the related work section, which is perfunctory — it could do a much better job positioning against the growing body of work on 3D-aware multimodal learning and object-centric representation learning. The ablation studies could also be deeper: how many slots are needed? What happens when the number of objects exceeds the slot count? How sensitive is the method to the quality of the pretrained backbone?

Verdict: weak accept — The core idea of adding spatial slots on top of pretrained omni-modal encoders is clean, the results are strong, and the dataset contribution adds value, though more rigorous analysis of failure modes and ablations would strengthen the case.

Takeaways

Three concrete things a practitioner can steal:

1. The “few extra tokens” pattern for extending frozen encoders. Instead of fine-tuning a massive pretrained model to add a new capability, learn a small set of specialized tokens that attend to the frozen features. This is applicable anywhere you want to extend a foundation model without catastrophic forgetting — add a few tokens for depth, for temporal reasoning, for whatever spatial axis you need.

2. Uncertainty-aware spatial attributes. When predicting spatial properties from ambiguous signals (especially audio), outputting a mean *and variance is cheap and prevents overconfident grounding. This pattern transfers directly to any task where the input signal has variable spatial precision — think room-scale audio, weak GPS signals, or imprecise text descriptions like “near the window.”

3. Two-level matching for retrieval. If your retrieval system currently uses a single global vector, consider adding an object-level alignment score. Even a simple version (match top-k objects between query and candidate, average their similarities) can dramatically improve discrimination between scenes that “look similar overall” but have different internal structure.