GaussianGPT: Autoregressive 3D Scene Generation via Token Prediction

Paper: 2603.26661 Authors: Nicolas von Lützow, Barbara Rössle, Katharina Schmid, Matthias Nießner Categories: cs.CV

The Gap

3D generative models have converged on diffusion and flow-matching: they treat scene generation as iterative refinement, denoising a holistic representation until it looks right. This works, but locks you into a specific generation paradigm — you can’t easily stop halfway, extend a scene spatially, or control generation temperature like you can with language models. The gap isn’t that diffusion fails; it’s that it doesn’t naturally support the compositional, incremental control that autoregressive models give you in other domains.

Problem: Diffusion = holistic refinement
         |
         v
Assumption: Sequential construction could enable finer control
         |
         v
Method: Compress 3D Gaussians -> discrete tokens -> GPT-style generation
         |
         v
Evidence: Comparable quality + completion/outpainting capabilities
         |
         v
Conclusion: Autoregressive is viable alternative for controllable 3D generation

The Increment

One sentence: Before this paper, 3D generation meant refining entire scenes at once; after, you can build scenes token-by-token with the same compositional control as text generation.

Core Mechanism

The method has three stages. First, a sparse 3D convolutional autoencoder compresses Gaussian primitives (position, scale, rotation, color, opacity) into a discrete latent grid using vector quantization — think of it as turning a cloud of 3D blobs into a vocabulary of spatial codes. Second, these codes are serialized into a sequence and fed to a causal transformer with 3D rotary positional embeddings, which learns to predict the next token given previous ones. Third, during generation, the transformer samples tokens autoregressively, which are decoded back into Gaussians for neural rendering.

Input Gaussians --> [Sparse 3D Conv Encoder] --> Continuous latents
                                                        |
                                                        v
                                                  [Vector Quantize]
                                                        |
                                                        v
                                                  Discrete tokens
                                                        |
                                                        v
                                              [Serialize to sequence]
                                                        |
                                                        v
                    [Causal Transformer + 3D RoPE] <-- Training
                                                        |
                                                        v
                                              Next-token prediction
                                                        |
                                                        v
                    Generated tokens --> [Decoder] --> Output Gaussians

Think of it like building with LEGO bricks. Diffusion methods are like sculpting clay — you start with a rough blob and gradually refine the whole thing until it looks right. GaussianGPT is like placing LEGO bricks one at a time: each brick (token) represents a chunk of spatial structure, and you decide where the next brick goes based on what you’ve already built. The autoencoder is your brick mold (turning arbitrary shapes into standardized pieces), the transformer is your building instructions (predicting which brick comes next), and the decoder snaps the bricks back into the final structure. Crucially, you can stop building anytime, extend in any direction, or change your building strategy mid-construction — things you can’t easily do when sculpting clay.

Key Concepts

• Vector Quantization (VQ): Imagine you have infinite shades of blue, but your printer only has 256 blue cartridges. VQ is the process of mapping each infinite shade to the nearest cartridge color. In this paper, continuous Gaussian features get mapped to the nearest entry in a learned codebook, turning smooth variations into discrete tokens. This discretization is what makes autoregressive modeling possible — transformers predict discrete symbols, not continuous values.

• 3D Rotary Positional Embedding (RoPE): Standard transformers use positional encodings to tell tokens where they are in a sequence. But 3D scenes aren’t just sequences — they have spatial structure in three dimensions. 3D RoPE extends the rotary embedding idea (which encodes position as rotation angles) to 3D coordinates, so the transformer knows not just “this token comes after that token” but “this token is spatially near that token in 3D space.” It’s like giving the model a compass that works in three dimensions instead of just along a line.

Framework Shift

Before (Diffusion):                  After (GaussianGPT):
                                     
  Noise --> [Denoise] --> Scene        Empty --> [Add token] --> [Add token] --> Scene
    ^           |                                    ^               ^
    |___________|                                    |_______________|
    (iterative refinement)                          (sequential construction)
    
  - Holistic updates                   - Incremental building
  - Fixed generation horizon           - Flexible stopping/extending
  - Hard to control locally            - Natural completion/outpainting

From parallel refinement to sequential construction, the core shift is trading holistic denoising for compositional token prediction.

Expert Assessment

Problem choice: Real gap. Diffusion’s dominance in 3D generation is partly inertia from 2D image success. The lack of controllable, compositional generation is a genuine limitation for interactive applications. This sits at the intersection of two mature ideas (autoregressive modeling, 3D Gaussians) rather than pushing either frontier, but the combination is underexplored.

Method maturity: Mostly engineering, not deep insight. The core idea — “apply GPT to 3D” — is straightforward. The technical contributions (sparse 3D conv, VQ, 3D RoPE) are competent adaptations of existing techniques. No obvious simpler approach is overlooked, but there’s no conceptual breakthrough either. It’s solid systems work.

Experimental integrity: Baselines seem fair (comparing to recent diffusion methods), but the paper would benefit from ablations on the VQ codebook size and serialization order. The qualitative results look good, but quantitative metrics (FID, etc.) aren’t deeply analyzed. The completion/outpainting demos are compelling but cherry-picked — no systematic evaluation of failure modes.

Writing quality: The abstract and intro are crisp. The method section gets bogged down in architectural details without enough intuition-building. The related work section is thorough but defensive (too much “unlike diffusion, we…”). Rewriting Section 3.2 to lead with the LEGO metaphor before diving into conv layers would help.

Verdict: weak accept — Solid execution of an underexplored idea with clear practical benefits, but limited conceptual novelty and incomplete experimental analysis.

Takeaways

The big steal: discretize-then-autoregressively-model is a pattern that transfers beyond 3D. If you’re working with continuous structured data (molecules, audio, video), consider whether VQ + causal transformer gives you controllability that diffusion doesn’t. Specifically, the ability to condition on partial structures (completion) and extend beyond training boundaries (outpainting) comes almost for free with autoregressive formulations.

Technical detail worth borrowing: 3D RoPE for spatially-structured sequences. If your tokens have natural 3D coordinates (voxels, point clouds, spatial graphs), standard positional encodings throw away geometric information. Extending RoPE to 3D is straightforward and gives the model spatial awareness.

Framing insight: The paper positions autoregressive modeling as “complementary” to diffusion rather than “better.” This is smart positioning for a paper introducing an alternative paradigm — it avoids overpromising while carving out a niche. If you’re proposing a non-mainstream approach, consider framing it as expanding the toolkit rather than replacing the incumbent.