GMOS: Grounding Moving Object Segmentation in 3D Space and Time

Paper: 2605.30352 Authors: Junyu Xie, Tengda Han, Weidi Xie, Andrew Zisserman Categories: cs.CV

The Gap

Moving Object Segmentation (MOS) methods have been stuck in a 2D+sequence paradigm: they rely on pre-computed optical flow or point trajectories (losing 3D geometry), and they treat “moving” as a video-level label (a car is either “moving throughout the video” or not), ignoring that objects start and stop moving at specific frames. This creates two concrete failures: (1) a parked car that drives away mid-video gets mislabeled for half its frames, and (2) depth ambiguities cause false positives when camera motion mimics object motion in 2D projections.

Prior work like OCLR and CaDeX use optical flow as input, which collapses 3D motion onto 2D image planes. MoCA-Mask uses point trajectories but still operates in 2D feature space. All treat motion as a sequence-level binary attribute, requiring objects to move consistently across the entire video to be detected.

Problem: 2D auxiliary inputs + sequence-level motion labels
   |
   v
Assumption: 3D geometry + frame-level motion state needed
   |
   v
Method: Direct RGB -> 3D scene representation -> per-frame motion queries
   |
   v
Evidence: +8.2% J&F on MOS benchmarks, 3.8x faster than multi-object baselines
   |
   v
Conclusion: Grounding in 3D space-time enables finer temporal control

The Increment

One sentence: Before GMOS, moving object segmentation required pre-computed 2D motion cues and treated motion as a video-level attribute; after GMOS, systems can segment moving objects directly from RGB with 3D awareness and frame-level temporal precision.

Core Mechanism

GMOS operates in three stages. First, a video encoder processes RGB frames into per-frame feature maps. Second, a 3D scene decoder lifts these 2D features into a volumetric representation using learned depth and camera pose estimation—this is where 3D grounding happens. Third, a motion query module samples this 3D volume at each frame and predicts per-object motion states (moving/static) and segmentation masks simultaneously.

The 3D scene decoder is the critical component. It doesn’t just stack 2D features; it explicitly models depth and camera motion to construct a consistent 3D representation across frames. When an object moves, its 3D position changes relative to this world-centric volume. The motion query module then asks: “At frame t, which 3D regions are changing position?” This question is answerable because the representation is grounded in 3D—camera motion is factored out, so only true object motion remains.

RGB Video (T frames)
   |
   v
[Video Encoder] --> 2D features per frame
   |
   v
[3D Scene Decoder] --> Volumetric representation
   |                    (depth + camera pose)
   v
[Motion Query Module] --> Per-frame motion state + masks
   |
   v
Output: \{mask_t, moving_t\} for each object at each frame t

Think of GMOS as a security system with depth-sensing cameras. Old systems (prior MOS methods) watch 2D video feeds and flag anything that moves on screen—but they can’t tell if motion is from an intruder walking or the camera panning. They also decide “this hallway has motion” for the entire night shift, missing when the intruder actually entered. GMOS builds a 3D map of the building first (the volumetric scene representation), then tracks which 3D positions change at each moment. Camera panning just rotates the viewpoint in the 3D map—it doesn’t create false alarms. And because it checks every frame independently, it knows the intruder entered at 2

AM, not just “sometime during the night.”

Key Concepts

• 3D Scene Grounding: Instead of analyzing motion in 2D image coordinates where camera movement and object movement are entangled, GMOS lifts features into a 3D volumetric space where camera pose is explicitly modeled. Imagine watching a car from a moving train—in your 2D view, the car appears to move even if it’s parked. But if you know the train’s motion and reconstruct the 3D scene, you can tell the car is stationary in world coordinates. GMOS does this computationally: it estimates depth and camera trajectory, then represents the scene in 3D where only true object motion (not camera-induced apparent motion) shows up as position changes across frames.

• Instantaneous Motion State: Prior MOS methods label objects as “moving” or “static” for an entire video sequence. But real objects transition—a person walks, stops, walks again. GMOS introduces per-frame motion queries that predict whether each object is moving at each specific frame. This is like the difference between saying “John exercised this week” (sequence-level) versus having a timestamped log showing John ran at 7 AM Monday, rested Tuesday, ran again Wednesday (frame-level). The latter enables precise temporal reasoning: you can ask “which objects were moving when the collision occurred?” rather than “which objects moved at some point in the video?”

• GMOS-2K Dataset: Existing MOS datasets only have sequence-level motion labels (object X moves in video Y). The authors manually annotated 2,210 videos from five VOS benchmarks with per-frame motion states—for each object, marking which frames it’s actually moving versus static. This is tedious but necessary: without ground truth for instantaneous motion, you can’t train or evaluate frame-level motion prediction. It’s like the difference between a movie’s genre tag (action film) versus scene-by-scene annotations (action in minutes 5-12, dialogue in 13-20). The latter enables fine-grained analysis.

Framework Shift

Before (mainstream approach):        After (this paper):

Video --> [Optical Flow] ----+       Video --> [3D Scene Decoder]
          [Point Tracks]  ---|           |
                |              |         v
                v              |     Volumetric Repr.
          2D Motion Features   |         |
                |              |         v
                v              |     [Motion Query]
          [Sequence Classifier]|         |
                |              |         +---> Per-frame
                v              |               motion state
          Video-level label    |               + masks
          (moving/static)      |
                               |
Key: Pre-computed 2D cues,    Key: Direct RGB input,
     sequence-level motion         3D grounding,
                                   frame-level motion

From 2D+sequence to 3D+instantaneous, the core shift is replacing auxiliary motion inputs with learned 3D scene structure that enables frame-level motion reasoning.

Expert Assessment

Problem choice: Real gap. Sequence-level motion labels genuinely limit temporal reasoning—you can’t answer “when did the object start moving?” with current methods. The 3D grounding addresses a known failure mode (camera motion confounds 2D motion cues) that practitioners encounter in robotics and autonomous driving. This sits at the intersection of MOS and 3D scene understanding, a natural evolution as video models mature.

Method maturity: The 3D scene decoder is the clever bit—it’s not just depth estimation bolted on, but a learned volumetric representation that factors camera motion. However, the motion query module is fairly standard transformer queries. The design is clean but not groundbreaking; the real contribution is showing that 3D grounding + frame-level queries work better than 2D flow + sequence labels. No simpler approach is obviously overlooked, though one could question whether full volumetric reconstruction is necessary or if depth maps alone suffice (the paper doesn’t ablate this thoroughly).

Experimental integrity: Baselines are fair—they compare against recent MOS methods (OCLR, MoCA-Mask, CaDeX) and unsupervised VOS methods. The new MOS-I protocol with three metrics (temporal precision, recall, F1) is well-motivated. However, the GMOS-2K dataset annotations are single-annotator with no inter-rater reliability reported, which is a weakness for a dataset contribution. The speed comparisons are honest (they’re faster than multi-object methods but slower than single-object variants). One red flag: the paper doesn’t show failure cases or discuss when 3D grounding fails (e.g., textureless scenes where depth estimation degrades).

Writing quality: The abstract and introduction are crisp. The method section is dense—Figure 2 (architecture diagram) tries to show too much at once and would benefit from being split into separate diagrams for each stage. The related work section is thorough but reads like a literature review rather than positioning the contribution. The experimental section is strong, but the ablation study (Table 4) is buried and should be elevated—it’s where the 3D grounding claim is actually validated. Rewriting Section 3.2 (3D Scene Decoder) with a concrete example walkthrough would make the method much more accessible.

Verdict: weak accept — Solid contribution with real-world applicability, but the method is more “careful engineering of known components” than a conceptual breakthrough, and the dataset annotation quality needs validation.

Takeaways

Steal the 3D grounding pattern: If you’re working on any video understanding task where camera motion confounds your signal (action recognition, video anomaly detection, sports analytics), consider lifting 2D features into 3D before reasoning. The key is explicit camera pose estimation—don’t just use depth maps, model the camera trajectory so you can factor it out.

Frame-level queries beat sequence-level labels: When your task has temporal structure (objects transition between states), don’t collapse it into a single label. Use per-frame queries and let the model learn temporal consistency implicitly. This applies beyond MOS—think video captioning (what’s happening *now vs. summarizing the whole clip) or audio event detection (when did the glass break, not just “glass broke in this recording”).

Dataset annotation strategy: The GMOS-2K approach—taking existing datasets and adding finer-grained annotations—is cheaper than collecting new data. If your field has coarse labels (image-level tags, video-level categories), consider whether frame-level or region-level annotations would unlock new capabilities. The annotation cost is real but often feasible for 2K samples.

问题：二维辅助输入 + 序列级运动标签
   |
   v
假设：需要三维几何 + 帧级运动状态
   |
   v
方法：直接 RGB -> 三维场景表示 -> 逐帧运动查询
   |
   v
证据：MOS 基准上 +8.2% J&F，比多物体基线快 3.8 倍
   |
   v
结论：在三维时空中定位能实现更精细的时间控制

RGB 视频（T 帧）
   |
   v
[视频编码器] --> 逐帧二维特征
   |
   v
[三维场景解码器] --> 体积表示
   |                   （深度 + 相机位姿）
   v
[运动查询模块] --> 逐帧运动状态 + 掩码
   |
   v
输出：每个物体在每帧 t 的 \{mask_t, moving_t\}

之前（主流方法）：                之后（本文方法）：

视频 --> [光流] --------+         视频 --> [三维场景解码器]
         [点轨迹] ------|             |
              |          |             v
              v          |         体积表示
        二维运动特征     |             |
              |          |             v
              v          |         [运动查询]
        [序列分类器]     |             |
              |          |             +---> 逐帧
              v          |                   运动状态
        视频级标签       |                   + 掩码
        (运动/静止)      |
                         |
关键：预计算二维线索，  关键：直接 RGB 输入，
      序列级运动              三维定位，
                              帧级运动