One Model to Rule All Vision Tasks: Unified Multimodal Generation for Computer Vision

Paper: 2607.06560 Authors: Xiaoyang Han, Jianhua Li, Kewang Deng, Zukai Chen, Xuanke Shi, Sihan Wang, Boxuan Li, Linyan Wang, Siyi Xie, Xin You Categories: cs.CV

The Gap

Modern multimodal models are great at understanding images and generating text about them. However, when you want to do something like segment an object, estimate depth, or predict camera pose, you usually need to bolt on a separate, task-specific “head” to the model. This creates a fragmented landscape: a powerful vision-language model for VQA and captioning, but a different, specialized model architecture for dense prediction tasks like segmentation or depth estimation. The fundamental limitation is that the “output space” for these spatial and geometric tasks is fundamentally different from the text tokens these models are trained on.

This paper addresses that gap by asking: what if we could make any vision task’s output look like something the model already knows how to generate—either text, an image, or a mixture of both?

[Problem]
    |
    v
[Diverse vision tasks exist]
    |
    v
[Each task requires a unique prediction head/architecture]
    |
    v
[Fragmented models, poor scalability]
    |
    * (Logical leap)
    v
[What if all tasks can be expressed as generation targets?]
    |
    v
[Convert tasks to instruction -> (text|image|mixed) response pairs]
    |
    v
[Train a single unified multimodal generator]
    |
    v
[Evidence: Match specialized models across tasks with one model]
    |
    v
[Conclusion: Unified generation is a scalable path for vision foundation models]

The Increment

One sentence: Before this paper, integrating dense spatial prediction tasks (segmentation, depth, normals) into a powerful vision-language model required specialized architectural additions; after this paper, those tasks are framed as generating images (or mixed text-image outputs) from natural language instructions within a single, unmodified model architecture.

Core Mechanism

The core idea is a reframing of the problem space. Instead of designing a network with a final layer that outputs, say, a 2-channel depth map, they train the model to generate an image that visually represents the depth map. The input is a natural language instruction (“Estimate the depth map for this image”), optionally combined with a “visual prompt” (e.g., a bounding box drawn on the image to specify a region of interest). The model’s output is not a tensor passed through a task-specific head, but rather the same autoregressive image generation process used for creating any other picture.

To make this work, they built the SenseNova-Vision Corpus. This is the key data engineering effort. They took annotations for many different tasks (bounding boxes for detection, pixel maps for segmentation, point maps for 3D) and converted them into the model’s native formats. A bounding box becomes a set of text coordinates in the output. A segmentation mask becomes a generated image with colored regions. A depth map becomes a generated grayscale image. Crucially, the *same instruction-response format is used for all of them. The model is then fine-tuned from a pretrained unified multimodal model on this corpus, mixed with other general multimodal data to preserve its core abilities.

[Input: Image + Instruction Text]
        |
        v
[Instruction Tokenizer & Image Encoder]
        |
        v
[Unified Multimodal Model (Transformer Backbone)]
        |                   |
        v                   v
[Text Generation Space] [Image Generation Space]
        |                   |
        +------- Merge ------+
                |
                v
[Output: (Text | Image | Text+Image) Response]
        |
        +----------------+------------------+
        v                v                  v
   [Symbolic         [Dense Spatial      [Compositional
    (e.g., bbox       Predictions         (e.g., "detect all
     coordinates,     (e.g., depth map,   red cars and show
     category)]        seg mask, normals)] their segmentation"]

The metaphor: Imagine a universal translator for visual information. The old way was like having a separate translator for each language pair (French-English, German-English, Japanese-English), each with its own dictionary and rules. This paper proposes building a single translator who already knows all the world’s languages (text, images). When you give them a complex request involving “visual languages” (like “draw me a map of the room’s surfaces in this photo” or “list the objects here and outline them”), they don’t pull out a special “depth-map” or “segmentation” dictionary. They use their universal understanding to *compose an answer in the native output formats they already know—sometimes just writing a description (text), sometimes drawing a picture (image), and sometimes doing both.

Key Concepts

• Visual Generation Space: Think of the model as having two primary ways of “speaking”: it can write (text) or draw (image). Instead of creating a third, special way of speaking for “segmentation output” or “depth output,” this paper forces all visual understanding tasks to be expressed through the model’s existing drawing or writing abilities. For example, the model doesn’t have a “depth head”; it simply generates a new picture where pixel brightness corresponds to depth, just as it would generate a picture of a cat.

• Visual Prompts: This is like giving the model specific instructions with your hands. Instead of just saying “segment the dog,” you can circle the dog on the input image with a red line (a visual prompt) and say “segment *this object.” The model learns to associate these drawn visual cues with the generation task. This makes the instruction space more flexible and powerful, mimicking how humans give precise, spatially-grounded instructions.

Framework Shift

Before (mainstream approach):               After (this paper):

+-------------------+                       +-------------------+
|    Input Image    |                       |    Input Image    |
+--------+----------+                       +--------+----------+
         |                                           |
         v                                           v
+-------------------+                       +-------------------+
|   Core Backbone   |                       |   Core Backbone   |
| (Vision-Language) |                       | (Unified Multimodal|
+--------+----------+                       |    Generator)     |
         |                                   +--------+----------+
         |                                            |
    +----+----+----+----+----+----+                    |
    |    |    |    |    |    |    |                    |
    v    v    v    v    v    v    v                    v
[Seg][Dep][Det][KP]...  [Text]              +-------------------+
 Heads Heads Head Head    Head               | Generation Space |
    |    |    |    |                          | (Text + Image)    |
    v    v    v    v                          +--------+----------+
  Task-specific outputs                              |
                                               +-----+-----+
                                               |           |
                                               v           v
                                        [Generate Text] [Generate Image]
                                               |           |
                                               v           v
                                        [Answers]  [Seg Mask, Depth Map, etc.]

From task-specific heads to a unified generation space, the core shift is treating every visual output as a generation modality rather than a specialized prediction.

Expert Assessment

Problem choice: This is a real and significant gap. The fragmentation between “understanding” models (VLMs) and “dense prediction” models (for segmentation, depth, etc.) is a major roadblock for building true, general-purpose vision foundation models. This paper sits squarely on the frontier of unifying these streams.

Method maturity: It’s a clever insight combined with significant data engineering force. The idea of reframing outputs as generation targets is elegant. However, the heavy lifting is done by the conversion of vast, disparate datasets into the unified instruction-response format. It’s more “clever framework + brute-force data curation” than a fundamentally new algorithmic component. Simpler approaches might exist, but they likely haven’t achieved this scale of task integration.

Experimental integrity: The baselines appear fair, comparing against strong, specialized models on standard benchmarks. The breadth of tasks covered (detection, segmentation, depth, normals, point maps, camera pose) is impressive and credible. A red flag might be the cherry-picking of qualitative results, as this method is inherently good at producing *visually plausible outputs that may mask fine-grained inaccuracies not caught by aggregate metrics.

Writing quality: The writing is clear and methodical. However, the “Method” section reads more like a data processing pipeline description. Rewriting it to better contrast the paradigm shift—spending less time on corpus statistics and more on the conceptual leap of mapping tasks to generation spaces—would elevate the paper’s impact and make the core contribution more memorable.

Verdict: Weak accept — It presents a compelling and practically significant framework for unifying vision tasks, backed by a massive data effort and broad experiments. While not algorithmically flashy, the reframing is valuable and the results are convincing enough to influence the field’s direction.

Takeaways

1. The “Everything is a Generation Target” Reframe: The most transferable idea is this: when you have a powerful generative model, consider if other seemingly non-generative tasks (classification, regression, detection) can be creatively reformulated as generating a specific, structured output in the model’s native space (text, image, or even code). This can collapse many problem-specific heads into one.
2. Visual Prompts as a Universal Control Mechanism: The technique of using drawn masks, points, or boxes as part of the instruction is a powerful and intuitive way to ground language in spatial reality. This pattern can be adopted in any multimodal system requiring fine-grained, spatially-aware control.
3. Data Unification is the Hidden Hero: The paper’s major contribution may not be the model, but the *SenseNova-Vision Corpus methodology. The systematic process of converting annotations from various CV tasks into a uniform instruction-response format is a blueprint for anyone looking to combine datasets from different sources and tasks for large-scale training. The bottleneck shifts from model design to data engineering and formatting.