MirrorWorld: Teaching Video Diffusion Models What Belongs in a Mirror

Paper: 2608.07463 Authors: Youjun Zhao, Alex Warren, Gary K. L. Tam, Rynson W. H. Lau Categories: cs.CV, cs.LG

Note up front: I’m working from the abstract and the framing it commits to. Where I’m inferring implementation details (which foundation model, how the transform is parameterized), I say so.

The Gap

Video diffusion models are now good enough that the interesting failures are the semantic ones. Mirrors are a beautiful example. A generative model can paint a mirror that looks like a mirror — right frame, right glassy sheen, right specular falloff — and put completely the wrong world inside it. The person standing in front of it isn’t in it. Or they are, but facing the wrong way, or at the wrong scale, or drifting frame to frame.

Three lines of prior work each stop short of this:

• Image-based reflection generation (the MirrorFusion / SynMirror family, depth-conditioned inpainting of a mirror plane) does get single-frame reflections roughly right, but has no notion of temporal coherence, and applying it per-frame makes the reflection flicker as its guesses re-roll.
• Video inpainting (flow-based methods like ProPainter/E2FGVI, and newer diffusion inpainters) is *extremely good at making a masked region look plausible and temporally smooth. That’s exactly the problem: it optimizes for local plausibility, and a mirror’s correct content is determined non-locally, by pixels somewhere else in the frame.
• Video mirror detection knows where mirrors are but never generates anything.

So the gap is a missing inductive bias: nothing in a standard VDM’s objective says “the tokens inside this region are a transformed function of tokens outside it.” The paper’s claim is that this relationship factorizes cleanly into a what and a how, and that each half needs its own supervision signal.

[ Problem ] content inside a mirror is determined by content outside it
     |
     v
[ Prior work and where it stops ]
   image reflection gen ..... one frame at a time . flickers in video
   video inpainting ......... locally plausible fill . globally wrong
   video mirror detection ... locates mirrors . generates nothing
     |
     v
[ Assumption ] reflection = WHAT is reflected  x  HOW it is arranged
     |          two separable sub-problems . two separate losses
     v
[ Method ] SRD: distill scene-to-mirror affinity from a frozen VFM
     |     GTA: learn a scene -> mirror spatial transform
     v
[ Evidence ] 4 existing video mirror datasets repurposed into a
     |        reflection reconstruction benchmark . beats image-based
     |        reflection methods and strong video inpainters
     v
[ Conclusion ] scene-to-mirror relation must be supervised explicitly
               rather than left to a generic generative prior

The Increment

One sentence: Before, filling a mirror region in video was treated as a hole-filling problem and the model guessed; after, it’s treated as a *correspondence problem with two explicit halves — a semantic link saying which scene content belongs inside, and a geometric link saying where it goes — plus the first benchmark that can actually measure whether you got it right.

Core Mechanism

The backbone is an ordinary video-diffusion inpainting setup: encode the video into latent tokens, mask the mirror region, denoise conditioned on the visible context and the mirror mask. Everything new lives in two auxiliary objectives that shape what the transformer’s internal attention does, rather than in new architecture on the generation path.

Semantic Relation Distillation (SRD) exploits the fact that a frozen self-supervised vision backbone (DINO-style) already produces features where a person and their reflection land close together — it’s an appearance-matching machine and nobody had to train it for mirrors. Run the teacher on the *ground-truth frames, compute the token-to-token affinity matrix between scene positions and mirror positions, and you get a soft table of “this patch inside the mirror corresponds to that patch outside it.” SRD pushes the diffusion transformer’s own attention distribution between mirror tokens and scene tokens toward that teacher table. The generator is thereby told to look at the right scene content while synthesizing the mirror, which is a much better-conditioned learning problem than “reconstruct these pixels.”

Geometric Transformation Alignment (GTA) handles the part semantics can’t: a planar mirror doesn’t just copy content, it applies a specific spatial map — a flip about the mirror plane composed with a perspective warp determined by the mirror’s pose relative to the camera. GTA learns that transform (I read it as a low-parameter warp, homography-like, predicted per scene/frame) and uses it to align scene features into the mirror region, supervising the arrangement of what SRD has already selected. So SRD constrains the attention’s *targets and GTA constrains its geometry.

 masked video + mirror mask
        |
        v
 +----------------------------+
 | VAE encode to latent tokens|
 +----------------------------+
        |
        v
 +--------------------------------------------------+
 |     Video Diffusion Transformer (trainable)      |
 |                                                  |
 |   scene tokens <--- attention A_s ---> mirror     |
 |        ^                                  ^      |
 +--------|----------------------------------|------+
          |                                  |
    SRD   | match affinity            GTA    | align layout
          |                                  |
 +--------+-----------+          +-----------+---------+
 | frozen VFM         |          | transform module T  |
 | (DINO-like)        |          | scene -> mirror     |
 | teacher affinity   |          | flip + warp params  |
 | A_t (i to j)       |          |                     |
 +--------------------+          +---------------------+
        WHAT                              HOW
          \                                /
           +---------> denoise <----------+
                          |
                          v
                  reflection-consistent video

The metaphor: a theater production with a second stage. The mirror is a second stage visible through an open doorway, and you need the doorway scene to stay consistent with the main stage all night.

• The pretrained VDM is the scenery crew. Superb painters. Give them a blank doorway and they’ll fill it with something gorgeous and internally coherent — and entirely unrelated to the play.
• SRD is the casting director. It doesn’t paint anything; it hands over a cast list that says *the figure in the doorway is the same person as the one at stage left. And crucially, it doesn’t invent that list — it borrows one from the frozen foundation model, the veteran script supervisor who already knows who’s who by sight, without ever having been briefed on this production.
• GTA is the blocking chart. Casting alone gives you the right actor in the wrong spot, facing the wrong way. The blocking chart is a fixed coordinate map from main stage to doorway stage: cross to this side, turn this direction, stand at this depth.
• The training loss on pixels is the audience. Necessary, but far too slow a signal to teach casting and blocking on its own — which is why the two notes are given directly to the crew during rehearsal rather than inferred from applause.

Retell it that way and you have the paper: casting plus blocking, both handed to painters who were already competent at painting.

Key Concepts

• The what/how factorization. “Generate a correct reflection” is one entangled objective, and gradient descent on pixel error gives you almost no traction on it. Split it and each half becomes tractable: *what is a matching problem (which scene region does this mirror region correspond to), how is a warping problem (given the correspondence, what spatial map relates them). Concretely: if a red-jacketed man stands left of a mirror, what says “red jacket, this man, not a generic person”; how says “he appears on the right side of the mirror, smaller, rotated toward the glass.” Getting one right and the other wrong produces two visibly different failure modes — right person in the wrong place versus wrong person in the right place — which is itself decent evidence the split is real.

• Attention distillation from a frozen backbone. An affinity matrix is just a big table of “how similar is patch i to patch j,” and self-supervised backbones like DINO produce features where that table is semantically meaningful for free: your face and your reflection score high, your face and the carpet score low. A diffusion transformer *also has such a table — its attention weights. SRD says: you already have a teacher who knows the answer to the correspondence question, so instead of hoping the student discovers it from reconstruction error, copy the table. This is a broadly reusable trick and not specific to mirrors: any generation task with a hidden long-range correspondence (shadows, symmetric objects, occluded completion) can borrow a teacher’s affinities.

• Reflection reconstruction as a proxy benchmark. There is no dataset of “scene with mirror, plus ground-truth-correct reflection you must invent.” So the authors manufacture one: take four existing video-mirror-*detection datasets, use their mirror masks to erase real reflections, and require the model to put them back. Now you have pixel-exact ground truth and can report PSNR/SSIM/LPIPS/FVD. Clean and cheap — but note what it measures. It rewards recovering a reflection that was physically present and is heavily constrained by context. It does not directly test placing a new mirror in a scene, which is arguably the downstream application people want.

Framework Shift

Before (mainstream inpainting)      After (MirrorWorld)

  scene                               scene
 [ ~~~~~~~~~~ ]                      [ ~~~~~~~~~~ ]
       |                               |        |
       | context only                  | what   | how
       | (local plausibility)          | (SRD)  | (GTA)
       v                               v        v
 [ mirror region ]                    [ mirror region ]
   fill with something                  transformed copy of
   that looks fine and                  identified scene content
   moves smoothly
                                     explicit typed link
 no typed relation                   scene -> mirror
 scene -> mirror

From hole filling to constrained correspondence: the core shift is that the mirror region stops being an unknown to hallucinate and becomes a known function of the rest of the frame, with the semantic and geometric halves of that function supervised separately.

Expert Assessment

Problem choice. Real gap, and well-timed. The field has largely finished the “make it look good” phase of video generation and is entering the “make it physically and semantically consistent” phase — mirrors are a crisp, checkable instance of that, in the same family as shadows, refraction and multi-view consistency. It’s a niche framed narrowly enough to be solvable and general enough to matter. The what/how decomposition is the paper’s best idea and would survive even if both proposed modules were replaced.

Method maturity. SRD is genuine cleverness: distilling correspondence from a frozen backbone into cross-attention is the right lever, cheap, and probably the part that transfers. GTA reads as the weaker half. A planar mirror’s transform is *derivable from geometry — estimate depth and the mirror plane normal and you can compute the warp in closed form, which is exactly the direction the depth-conditioned image work went. Learning it end-to-end from a small repurposed dataset risks getting a soft, blurry approximation of something a geometry estimator hands you exactly. I’d want an ablation against “compute the reflection warp analytically and use it as hard conditioning,” and I’d want to know how GTA degrades on curved or multi-planar mirrors, or when the mirror plane is near-parallel to the view direction. Also unaddressed in the abstract: a very strong dumb baseline — copy the scene, flip it, warp it with an off-the-shelf homography, and hand that to the inpainter as a hint image. If MirrorWorld doesn’t clearly beat that, the machinery is hard to justify.

Experimental integrity. The baseline set is reasonable but structurally favorable. Image-based reflection methods applied per-frame *will flicker; beating them on video metrics is close to definitional. Generic video inpainters aren’t trained for this task at all. The honest comparison is against the same backbone fine-tuned on the same benchmark without SRD/GTA, and against the analytic-warp baseline above — the abstract’s “representative image-based methods and strong video inpainting baselines” doesn’t tell me those exist. Second flag: on a reconstruction task with pixel ground truth, PSNR/SSIM systematically reward conservative, slightly-blurred output, so a method that produces smoother reflections can win the table while looking worse to a human. A user study or an identity-consistency metric (does the reflected face match the real face under a recognition embedding) would be far more persuasive than another decimal of PSNR. Third: mirror regions in detection datasets vary wildly in size, and metrics computed over the full frame will be dominated by unmasked context — I’d want masked-region-only numbers, stratified by mirror area.

Writing quality. The abstract is unusually disciplined — the what/how framing is stated once and then used to organize everything, which is how it should be done. The section I’d bet is thinnest is the benchmark construction: repurposing four detection datasets into a generation benchmark involves a pile of consequential choices (mask dilation, how much of the mirror to erase, how splits avoid scene leakage between train and test, whether tiny mirrors are filtered) and papers routinely compress that into one paragraph. Rewriting it properly, with per-dataset statistics and masked-region metrics, would raise the whole contribution from “our method wins” to “here is the task, here is how to measure it, and by the way our method wins.” The benchmark is likely the more durable contribution; it deserves more space than the modules.

Verdict: weak accept — the decomposition and the distillation trick are genuinely useful and the benchmark fills a real void, but the geometric half looks like it’s learning something that could be computed, and the reported comparisons as framed don’t yet rule out a much simpler pipeline.

Takeaways

• When a generative task hides a long-range dependency, don’t hope the model finds it — distill the correspondence from a frozen self-supervised backbone into the generator’s attention. DINO-family features already encode “these two distant patches are the same thing.” That teacher affinity matrix is free supervision, and the recipe transfers directly to shadow synthesis, symmetric-object completion, occlusion inpainting, and multi-view consistency.
• Factorize entangled objectives into “which” and “where.” Pixel loss on an entangled target gives weak gradients. Split content selection from spatial arrangement, give each its own loss, and you get both better learning signal and interpretable failure modes (wrong content vs. wrong placement) that tell you which half to fix.
• You can manufacture a generation benchmark out of a detection dataset. Masks meant for segmentation become masks for inpainting, and the erased region becomes exact ground truth. Cheap way to bootstrap evaluation for any “generate content that should be consistent with context” task — just be explicit that reconstruction is a proxy for synthesis, not the same thing.
• Reverse-engineer their weakness for your own work: if part of your task has a closed-form physical solution (a reflection warp, a projection, a known symmetry), compute it and condition on it instead of learning it. Spend learned capacity on the part that isn’t derivable.