Omni123: A Single Autoregressive Model for Both 2D Images and 3D Generation

Paper: 2604.02289 Authors: Chongjie Ye, Cheng Cao, Chuanyu Pan, Yiming Hao, Yihao Zhi, Yuanming Hu, Xiaoguang Han Categories: cs.CV, cs.AI

The Gap

The paper is going after a pretty specific bottleneck in current 3D generation.

We already have strong native multimodal models for text and images: one model, one token space, many tasks. But 3D has mostly lagged behind for two reasons. First, there just is not enough high-quality 3D data. Second, a lot of current text-to-3D systems are not really “3D-native” in the same sense that modern text-image models are native. They often work by generating or editing images first, then lifting those results into 3D through optimization, score distillation, multi-view fitting, or reconstruction stages. That works surprisingly often, but it is a brittle workaround.

The abstract is clearly positioning against those indirect pipelines: methods that use strong 2D priors and then recover 3D through optimization. The problem with that family is familiar: geometry is under-constrained, multi-view consistency is fragile, and the system inherits artifacts from the image model rather than learning a unified 3D-capable representation. In other words, existing methods can produce 3D outputs, but they do not really internalize 3D as a first-class modality.

So the gap is not just “better 3D quality.” It is this:

1. Can you build a 3D-native foundation model instead of a 2D-first lifting pipeline?
2. Can you do it without requiring massive aligned text-image-3D triplets, which basically do not exist at scale?
3. Can abundant 2D data be used not merely as supervision for rendering, but as a structural prior for 3D?

That is the hole Omni123 tries to fill. The move is conceptually elegant: if images and 3D describe the same object from different angles of abstraction, maybe consistency across those modalities can itself serve as the missing geometric regularizer.

[Problem]
limited 3D data
indirect 2D_to_3D pipelines
weak geometry consistency
        |
        v
[Assumption]
image and 3D consistency
can act as implicit structure
        |
        v
[Method]
shared token space for
text image 3D
+
interleaved X_to_X training
on heterogeneous pairs
+
semantic visual geometric cycles
        |
        v
[Evidence]
better text_guided 3D generation
better editing
improved multi_view consistency
        |
        v
[Conclusion]
2D abundance can regularize 3D
inside a single 3D_native
autoregressive foundation model

The Increment

One sentence: Before this paper, text-to-3D mostly borrowed intelligence from image models and reconstructed upward; after this paper, there is a clearer recipe for learning text, images, and 3D jointly in one autoregressive model even when fully aligned 3-way data is scarce.

Core Mechanism

At the center of Omni123 is a unification trick: text, images, and 3D are all represented as discrete tokens in one sequence space, so a single autoregressive model can learn to continue sequences across modalities. Instead of saying “this is a text model plus an image model plus a 3D reconstructor,” the paper says: treat all three as different dialects of the same token language. Then text-to-image, text-to-3D, image-to-3D, 3D-to-image, and editing tasks become different prompting patterns for the same next-token predictor.

The important part is how they train with incomplete data. They do interleaved X-to-X training over heterogeneous paired datasets, meaning they do not require every example to have text, image, and 3D all at once. Some samples may be text-image, some image-3D, some text-3D. By interleaving different cross-modal tasks in sequence form, the model can stitch together a shared representation indirectly. This is the practical answer to the data scarcity problem: instead of waiting for perfect triplets, exploit partial alignments and let the model triangulate.

The other key ingredient is the use of semantic-visual-geometric cycles inside autoregressive sequences. The abstract gives paths like text to image to 3D to image. That matters because it introduces consistency pressure from several directions at once. Text should preserve semantics, image should preserve appearance cues, and 3D should preserve view-consistent geometry. If the model can traverse these loops reliably, then 2D data is no longer just extra supervision; it becomes a scaffold that helps shape better 3D structure.

[text tokens] ---> [shared tokenizer space] ---> [autoregressive transformer] ---> [image tokens]
      |                     |                           |                           ^
      |                     |                           |                           |
      |                     |                           v                           |
      |                     |                      [3D tokens] ---------------------|
      |                     |
      v                     v
[paired datasets]     [interleaved X_to_X sequences]

[text image]
[image 3D]
[text 3D]

training paths:
text > image
text > 3D
image > 3D
3D > image
text > image > 3D > image
text > 3D > image

A load-bearing metaphor here is a three-language translation school.

Imagine you are training one translator who must work across English, sketches, and CAD blueprints. The problem is that you have tons of English-sketch pairs, a modest number of sketch-blueprint pairs, and very few perfectly aligned English-sketch-blueprint triplets. The usual approach would be: translate English into sketches really well, then try to infer blueprints afterward. That gets you plausible drawings, but shaky engineering.

Omni123 instead trains one multilingual translator with a shared internal phrasebook. Text is one language, images are another, 3D is the third. The “interleaved X-to-X” setup is like giving the student mixed homework: sometimes English to sketch, sometimes sketch to blueprint, sometimes blueprint back to sketch. The cycle text to image to 3D to image is like asking the student not just to translate, but to round-trip the meaning and preserve what matters at each step. If the sketch changes too much after passing through the blueprint, you know the blueprint translation lost structure. That is exactly how the model uses 2D abundance to discipline 3D learning.

Key Concepts

• Shared discrete token space:
  Foundation models are good at sequence modeling. So if you can turn text, images, and 3D into token sequences, one model can learn all of them with the same machinery. Think of turning very different media into LEGO bricks of the same connector type. The bricks still mean different things, but now one machine can assemble them in a common process. For example, text tokens might say “a red chair,” image tokens encode how that chair looks in pixels, and 3D tokens encode its shape. Once all are sequences, generating 3D from text becomes “continue this mixed-modality sentence.”

• Cross-modal consistency as implicit supervision:
  If you do not have enough 3D labels, you can still force 3D to make sense by checking whether it agrees with images and text. This is like learning the shape of a mug not from a perfect CAD database, but from many photos and captions that all have to line up. If the generated 3D mug renders into views that stop matching the image side of the training signal, the model gets corrected. The geometry is not supervised directly at every point, but it is constrained indirectly by having to stay compatible with other modalities.

• Interleaved X-to-X training:
  Instead of training separate systems for separate tasks, you mix tasks into one stream. Sometimes the model sees text and predicts image; sometimes image and predicts 3D; sometimes a longer chain. This matters because the model does not learn “task A” and “task B” in isolation. It learns that all these tasks are connected projections of one latent world. A concrete analogy: rather than taking separate classes in speaking, listening, and writing, you do integrated language drills where each skill supports the others.

Framework Shift

Before (mainstream approach):        After (this paper):

[text]                               [text]
  |                                    |
  v                                    v
[2D image model]                     [shared token space]
  |                                    |
  v                                    v
[optimization lift]                 [one autoregressive model]
  |                                  /   |    \
  v                                 v    v     v
[3D asset]                       [image] [3D] [mixed cycles]
  |
  v
[render check]

text and image do the heavy         text image and 3D are co trained
lifting first                       in one sequence framework
3D is recovered later               with cross modal round trips

From 2D-first reconstruction pipelines to joint autoregressive world modeling across text, image, and 3D, the core shift is treating cross-modal consistency as the training signal that makes scarce 3D data go further.

Expert Assessment

Problem choice: This is a real gap, not a manufactured one. The field has gotten very good at making 3D by leaning on 2D diffusion and optimization, but that recipe is hitting a conceptual ceiling. If people want actual multimodal world models rather than fancy lifting tricks, they need native 3D representations trained under data scarcity. This paper sits right on that transition.

Method maturity: The central insight is clever rather than purely brute force: use shared tokenization plus heterogeneous cross-modal training to squeeze more value out of limited 3D data. That said, this is still a “system paper” in spirit. A lot depends on tokenizer quality, sequence design, and exactly how 3D is discretized. If those pieces are weak, the grand unification story can collapse into engineering fragility. So the idea is strong, but the maturity depends heavily on implementation details that the abstract alone cannot verify.

Experimental integrity: Based on the abstract, the claims are plausible and appropriately scoped: improved text-guided 3D generation and editing, not “solved 3D.” The main thing I would want to inspect closely is baseline selection. For a paper like this, it is easy to compare against older lifting pipelines without fully matching data scale, compute budget, or editing setup. I would also want strong multi-view consistency diagnostics, not just cherry-picked visuals. No obvious red flag from the abstract, but this is exactly the kind of paper where qualitative examples can oversell if the evaluation suite is soft.

Writing quality: The paper seems to have a solid high-level story, but the risk is vagueness around the hardest part: what exactly are the 3D tokens, and how do cycles operationally enforce geometry rather than just encourage modality agreement? If I could rewrite one section, it would be the method section around sequence construction and tokenization. That is the section that determines whether readers see a real foundation-model contribution or just a loosely unified training recipe.

Verdict: weak accept — the problem is important and the framing is genuinely fresh, but the paper will live or die on whether the empirical section proves this is more than a neat unification narrative.

Takeaways

A practitioner can steal several concrete ideas here:

1. Use partial pairings instead of waiting for perfect multimodal triplets.
   If your domain has text-image and image-structure data but almost no text-image-structure triples, you can still train a joint model by interleaving tasks.

2. Make cycles do regularization work.
   Round-trip paths like A to B to C to B are a practical way to impose consistency when direct supervision is sparse. This idea should transfer well to robotics, medical imaging, remote sensing, and audio-video-language setups.

3. Treat abundant modalities as priors for scarce ones inside the model, not only in post-processing.
   That is the subtle but useful reframing. Do not just use rich modalities to initialize or guide optimization afterward; bake them into training so the scarce modality inherits stronger structure.

4. Unified sequence modeling is most compelling when the tasks actually constrain each other.
   The lesson is not “tokenize everything because transformers like tokens.” The real payoff comes when one modality can catch another modality’s mistakes.