UniD: Teaching a Single Video Model Eight Scene Skills from Separate Textbooks

Paper: 2607.21592 Authors: Yihong Sun, Seoung Wug Oh, Jiahui Huang, Bharath Hariharan, Joon-Young Lee Categories: cs.CV

The Gap

Scene understanding isn’t one problem — it’s eight overlapping problems (depth, normals, segmentation, boundaries, human parts, albedo, shading, materials). Each problem has its own dataset, annotated by its own community, with zero overlap. The dream is a single model that predicts all eight from one forward pass.

Existing unified systems take two paths, both flawed:

1. Co-annotated data only: Train only on frames where *all eight labels exist. This is extremely rare, expensive to create, and tiny in scale. Model starves.

2. Pseudo-labeling: Run eight specialist models on unlabeled video to generate “fake” labels for everything. Computationally explosive. Noise accumulates. Quality ceiling is low.

The gap: How do you train a unified model from disjoint datasets without requiring annotation overlap or paying the pseudo-labeling tax?

UniD’s answer: let per-task experts (already trained on their own datasets) teach a single backbone through lightweight projectors, and rely on the pretrained diffusion model’s visual priors to stitch the knowledge together.

                    Problem
                       |
       8 dense predictions needed simultaneously
                       |
               +-------+-------+
               |               |
        Co-annotated       Pseudo-labeling
        (scarce, small)    (expensive, noisy)
               |               |
               +-------+-------+
                       |
            Insight: diffusion priors
            bridge disjoint domains
                       |
               +-------v-------+
               | UniD: experts  |
               | distill into   |
               | unified backbone|
               +-------+-------+
                       |
            Evidence: competitive accuracy,
            strong OOD generalization,
            temporal consistency
                       |
                   Conclusion

The Increment

One sentence: Before this paper, training a unified dense prediction model meant either starving it on tiny co-annotated datasets or burning compute on noisy pseudo-labels — after this paper, you can just let your existing experts teach a diffusion-backed backbone through cheap projectors and get competitive results on combinations never seen during training.

Core Mechanism

UniD has three layers:

Layer 1 — Per-task experts (frozen, pre-trained). Each expert is a specialist model already trained on its own domain-specific dataset (e.g., a depth estimator trained on depth-only data, a segmentation model on segmentation-only data). These experts stay frozen — they are the “teachers.” No cross-task data is needed.

Layer 2 — Lightweight task projectors. For each task, a small projection head maps the unified backbone’s features into that task’s output space. These are the “translation layer” — they convert the generalist’s representation into something the specialist can grade. They are cheap to train.

Layer 3 — Unified diffusion backbone (the student). A pretrained video diffusion model serves as the shared backbone. During training, it processes video frames and tries to produce features that, when passed through each projector, match the expert’s output. The diffusion model’s strong visual priors (learned during pretraining on massive image/video data) are what allow it to generalize across task domains without seeing co-annotated data.

Training loop per iteration:

1. Sample a video frame from any task-specific dataset.
2. Run it through the unified backbone.
3. Pass backbone features through the relevant task projector.
4. Compute distillation loss between projector output and frozen expert output.
5. Update backbone + projector (not the expert).

[Video Frame]----->[Unified Diffusion Backbone]----+---->[Projector 1]---->[Expert 1 (frozen)]
                         |                          |
                         |                          +---->[Projector 2]---->[Expert 2 (frozen)]
                         |                          |
                         |                          +---->[Projector K]---->[Expert K (frozen)]
                         |                                     |
                      (updated)                          (lightweight,
                                                         updated)
                         |
                    Distillation loss:
                    match expert output
                    for whichever task
                    the current sample belongs to

Structural metaphor — The University Tutorial System:

Imagine a university that wants to produce a polymath — one person who masters physics, art history, and molecular biology. But the university has a constraint: each department’s textbooks are completely incompatible (physics books have no art, biology books have no physics). Traditionally, you’d either write one mega-textbook covering all three (expensive, shallow) or hire eight research assistants to generate fake cross-disciplinary notes (noisy, slow).

UniD’s approach is different:

• The polymath student (unified backbone) already has a world-class general education (diffusion pretraining) — they understand images, textures, geometry intuitively.
• Each department keeps its own professor (frozen expert) with its own textbooks (domain-specific dataset).
• Instead of reading each professor’s textbook directly, the student attends tutorial sessions (task projectors) where the professor says: “Given what you already know, here’s how you’d frame this in my field.”
• The professor grades the student’s attempt, and the student adjusts.

The key insight: because the student’s general education is so strong (diffusion priors), they can learn from each department’s feedback separately and still combine the knowledge coherently. No cross-department textbook needed. No need for the physics professor and art professor to co-teach.

At exam time (inference), the student walks into any room, applies the relevant tutorial notes (projector), and produces expert-level work — including in rooms they never visited during training.

Key Concepts

• Disjoint training data: Imagine you’re learning to cook from eight different recipe books, but each book only covers one cuisine — no recipe appears in two books. “Disjoint” just means the datasets don’t overlap. Each dataset has labels for one task only. The challenge is learning a single model that can handle all tasks despite never seeing a single sample labeled for all tasks simultaneously. This is realistic because in the real world, depth datasets, segmentation datasets, and material datasets are collected by different teams for different purposes.

• Distillation from experts: Think of it like this: instead of reading a textbook yourself, you sit with someone who already read it and ask them to quiz you. The “expert” model has already learned the task from its dataset. During UniD training, the unified model tries to produce outputs that match what the expert would produce. This is much cheaper than pseudo-labeling an entire dataset upfront — you only compute expert outputs on-the-fly for the current training batch.

• Diffusion model priors: A diffusion model trained on millions of images has internalized an enormous amount of visual knowledge — how surfaces reflect light, how objects occlude each other, how textures flow along surfaces. This is the “general education” that makes everything else work. Without these priors, a random backbone would struggle to bridge the gap between, say, a depth-only dataset and a material-only dataset. With these priors, the backbone already “understands” scenes in a general way, and the distillation just needs to steer that understanding toward specific predictions.

Framework Shift

Before (mainstream approach):         After (this paper):

  +-------------------+               +-------------------+
  | Co-annotated data |               | Depth dataset     |
  | (all 8 labels per |               | Segmentation data |
  |  frame, rare)     |               | Normal data       |
  +--------+----------+               | Material data ... |
           |                          +--------+----------+
           v                                   |
  +--------+----------+               +--------v----------+
  | Single unified    |               | Per-task experts  |
  | model trained     |               | (frozen, each on  |
  | end-to-end        |               |  own data)        |
  +-------------------+               +--------+----------+
                                               |
                                      +--------v----------+
                                      | Lightweight       |
                                      | projectors        |
                                      +--------+----------+
                                               |
                                      +--------v----------+
                                      | Unified backbone  |
                                      | (diffusion model) |
                                      +-------------------+

From requiring all labels on the same frame to requiring none, the core shift is replacing co-annotation with expert distillation anchored by diffusion priors.

Expert Assessment

Problem choice: This is a real, practical gap. Unified dense prediction is genuinely bottlenecked by data annotation. The field has been circling this problem for years — either restricting to co-annotated data (NYU-VIP, etc.) or pseudo-labeling at scale (OmniModel, etc.). This paper names the problem clearly and proposes an elegant workaround. It sits at the intersection of multi-task learning and foundation model distillation, which is a hot and important trajectory.

Method maturity: Clever, not brute force. The insight that diffusion priors can bridge domain gaps from disjoint training is the real contribution — the architecture itself (backbone + projectors) is simple. One concern: the approach still requires pre-trained per-task experts, which means you’re not truly “starting from scratch” — you’re shifting the training cost upstream. But this is a reasonable engineering tradeoff. I don’t see an obviously simpler approach being overlooked.

Experimental integrity: The baselines are fair — they compare against per-task specialists and multi-task baselines on standard benchmarks. The OOD generalization experiments are particularly compelling because they test on task-dataset combinations never seen during training. One flag: the paper claims “competitive performance against per-task specialists,” but the gap is non-trivial for some tasks (materials, shading). The temporal consistency results are strong and hard to argue with. Overall, the numbers hold up.

Writing quality: Solid, but the method section could be tighter. The distillation procedure is explained clearly, but the choice of which expert architectures to use and why feels glossed over. The related work section is thorough. The video results on the project page are genuinely impressive and worth watching — the paper undersells them.

Verdict: weak accept — the core insight (diffusion priors bridge disjoint domains) is valuable and generalizable, though the architectural novelty is modest.

Takeaways

Three concrete things to steal:

1. “Expert distillation as data unification”: If you have multiple models trained on incompatible datasets, you can train a new unified model by having each expert supervise it on-the-fly. This is cheaper than pseudo-labeling entire datasets upfront. Transfer this to any multi-task setting where annotation is fragmented.

2. Diffusion priors as domain bridges: If your multi-task training data comes from wildly different distributions, a pretrained diffusion backbone may provide enough shared visual grounding to make it work. This insight extends beyond dense prediction — any vision task with fragmented data could benefit.

3. The projector pattern: Instead of modifying a shared backbone’s architecture for each task, keep the backbone fixed and add tiny per-task projection heads. This is embarrassingly simple but effective, and it keeps inference modular — you can swap tasks by swapping projectors.