C4G: Compact Gaussian Queries for Globally Coherent 4D Scene Reconstruction

Paper: 2605.31595 Authors: Mungyeom Kim, Minkyeong Jeon, Honggyu An, Jaewoo Jung, Hyuna Ko, Jisang Han, Hyeonseo Yu, Donghwan Shin, Sunghwan Hong, Takuya Narihira Categories: cs.CV

The Gap

Existing feed-forward 4D reconstruction methods (like DynaMoN, SC-GS) predict 3D Gaussians independently for each pixel in each frame. This creates two problems: (1) the same physical point gets represented by different Gaussians across frames, causing duplication and inconsistency, and (2) each Gaussian only sees local features from its source pixel, leading to view-dependent biases that break down when the camera moves significantly or time gaps are large. The field has feed-forward speed but lacks global motion coherence.

Problem: Per-pixel Gaussian prediction
    |
    v
Assumption: A compact set of learnable queries can aggregate
            temporal context and model global motion
    |
    v
Method: Timestamp-conditioned Gaussian query tokens
        + temporal feature aggregation
        + position modulation by time
    |
    v
Evidence: Fewer Gaussians, better novel view synthesis,
          robust to large temporal gaps
    |
    v
Conclusion: Global motion modeling > per-pixel prediction

The Increment

One sentence: Before C4G, feed-forward 4D reconstruction meant predicting Gaussians pixel-by-pixel per frame; after C4G, a fixed set of learnable query tokens aggregates features across time and decodes motion-aware Gaussians, achieving global coherence without per-scene optimization.

Core Mechanism

C4G uses a fixed set of learnable Gaussian query tokens (think of them as persistent “slots” that represent the scene). For a given target timestamp, each query token attends to features across all frames in the video, aggregating information about what’s happening at different times. The token then decodes into a 3D Gaussian—position, rotation, scale, opacity, color—but crucially, the position is modulated by the target timestamp through a learned motion field. This means the same query token can represent a moving object at different positions depending on when you ask.

Input: Video frames [F1, F2, ..., Fn] + target time t
    |
    v
[Feature Extractor] --> Frame features [f1, f2, ..., fn]
    |
    v
[Query Tokens Q1, Q2, ..., Qk] (learnable, fixed count)
    |
    +---> [Cross-attention over all frame features]
    |         |
    |         v
    |     Aggregated temporal context per query
    |
    +---> [Decode to Gaussian parameters]
              |
              v
          (pos_base, rot, scale, opacity, color)
              |
              +---> [Position Modulation by time t]
                        |
                        v
                    pos(t) = pos_base + motion_field(t)
                        |
                        v
                    Render from target viewpoint

Think of it like a theater production. Traditional methods assign an actor (Gaussian) to every seat in the audience (pixel) for every scene (frame)—you end up with thousands of actors, many playing the same character inconsistently. C4G instead has a fixed troupe of actors (query tokens). Each actor watches all rehearsals (cross-attention over frames), learns their character’s arc across the entire play (temporal aggregation), and knows where to stand at any given moment (timestamp-conditioned position). When you ask “show me minute 3,” each actor moves to their position at that timestamp. The stage (scene) is coherent because the same actors are tracking the same roles throughout.

Key Concepts

• Gaussian Query Tokens: These are learnable vectors that act as persistent identifiers for scene content. Unlike per-pixel predictions that create new Gaussians for every frame, query tokens are shared across the entire video. Each token “owns” a piece of the scene and is responsible for tracking it over time. During training, the tokens learn to specialize—one might track a person’s torso, another a background wall. The key insight: by having a fixed, compact set of queries, you force the model to learn consistent representations rather than generating redundant, frame-specific Gaussians.

• Timestamp-Conditioned Position Modulation: A 3D Gaussian has a center position in space. For static scenes, this position is fixed. For dynamic scenes, C4G decodes a base position from each query token, then adds a time-dependent offset learned by a motion field network. Feed the motion field a timestamp, it outputs a displacement vector. This lets the same Gaussian represent a moving object: at t=0 it’s at position A, at t=5 it’s at position B. The motion field is learned end-to-end, so the model discovers how objects move without explicit tracking.

• Cross-Temporal Feature Aggregation: Each query token doesn’t just look at one frame—it attends to features from all frames simultaneously using cross-attention. This is like giving each query a “memory” of the entire video. If a person walks behind a tree in frame 10, the query token representing that person has already seen frames 1-9 and 11-20, so it knows the person exists even when occluded. This global context is what enables robustness to large temporal gaps and view changes.

Framework Shift

Before (per-pixel prediction):        After (C4G):

Frame 1 --> [CNN] --> Pixel grid      Video [F1...Fn] --> [Feature Extractor]
              |                                |
              v                                v
        [Predict Gaussian                 Frame features
         per pixel]                            |
              |                                v
              v                           [Query Tokens Q1...Qk]
        Gaussians_1                       (fixed, learnable)
                                               |
Frame 2 --> [CNN] --> Pixel grid              v
              |                           [Cross-attention
              v                            over all frames]
        [Predict Gaussian                      |
         per pixel]                            v
              |                           Aggregated context
              v                                |
        Gaussians_2                            v
                                          [Decode + Time
(Duplicates, no                            Modulation]
 temporal coherence)                           |
                                               v
                                          Gaussians(t)
                                          (compact, coherent)

From scattered per-frame predictions to persistent queries with global temporal awareness, the core shift is replacing spatial locality with temporal identity.

Expert Assessment

Problem choice: Real gap. Feed-forward 4D reconstruction is a hot area (DynaMoN, SC-GS, 4DGS all published recently), and the duplication/view-dependence issue is well-documented. The problem sits at the intersection of efficiency (feed-forward) and quality (motion coherence), which matters for practical applications like AR/VR content creation.

Method maturity: The query token idea is borrowed from DETR-style object detection and adapted cleverly here. The timestamp-conditioned position modulation is straightforward but effective. However, the paper adds a video diffusion model for rendering enhancement, which feels like a patch—if the core method worked perfectly, you wouldn’t need a separate diffusion model to fix details. The motion field is learned implicitly; an explicit motion prior (optical flow, depth) might be simpler and more interpretable.

Experimental integrity: Baselines are fair (DynaMoN, SC-GS, Deformable-GS). The paper shows strong PSNR/SSIM numbers and uses significantly fewer Gaussians (10k vs 100k+), which is impressive. However, the diffusion enhancement module is only evaluated qualitatively—no ablation on how much it contributes to the final metrics. The claim about “not requiring camera poses” is misleading; the method still needs camera intrinsics and relative poses for rendering, just not absolute poses. Red flag: no user study or perceptual metrics beyond LPIPS.

Writing quality: The abstract and intro are clear. The method section is dense—Figure 2 tries to show everything at once and ends up cluttered. The motion field formulation (Section 3.2) could be explained more intuitively before diving into equations. The related work section is thorough but reads like a list. If the authors rewrote Section 3.3 (diffusion enhancement) to justify why it’s necessary rather than just describing it, the paper would feel more cohesive.

Verdict: weak accept — Solid contribution with clear improvements over baselines, but the reliance on a diffusion model for final quality and lack of perceptual evaluation weaken the claim that the core method alone is sufficient.

Takeaways

Learnable query tokens for temporal consistency: If you’re building any system that processes sequences (video, time-series, multi-frame data), consider using a fixed set of learnable queries instead of per-frame predictions. The queries force the model to learn persistent representations, reducing redundancy and improving coherence. This pattern transfers beyond vision—think audio source separation, multi-agent tracking, or even document understanding across pages.

Timestamp as a conditioning variable: Instead of predicting separate outputs for each time step, decode a base representation and modulate it with time. This is cheaper (one decoder, many timestamps) and more coherent (smooth interpolation between frames). Applicable anywhere you have temporal data: animation, forecasting, simulation.

Cross-attention for global context: When local receptive fields aren’t enough (occlusions, large gaps), use cross-attention to let each element see the full context. The cost is quadratic in sequence length, but for moderate-length videos or bounded contexts, it’s worth the coherence gain.