FlowMimic: Bootstrapping Video Editing Data from Image Edits via Pixel-Pair Warped Flow

Paper: 2607.18227 Authors: Dingyun Zhang, Lixue Gong, Wei Liu Categories: cs.CV

The Gap

Video editing research has a dirty secret: the data pipeline is a nightmare. Collecting high-quality video editing pairs typically involves manually annotating object masks across frames, synthesizing paired data via image-to-video models and ControlNet-style guidance (which introduces its own errors), then running VLM-based quality filters to salvage what’s usable. This pipeline is labor-intensive, fragile, and—crucially—it scales poorly to new editing tasks. As a result, the diversity of video editing tasks available for training lags far behind what image editing models enjoy.

Image editing, by contrast, is relatively mature: large-scale paired datasets exist, covering style transfer, object replacement, attribute modification, and more. The core question this paper asks is: can we harvest all that image editing richness and directly convert it into video editing data, without the painful manual pipeline?

A second, related gap: existing video editing models lean heavily on external crutches at inference time. They either fine-tune an extra multimodal LLM to parse instructions, or they require explicit mask sequences as auxiliary input to tell the model where to edit. Neither approach internalizes the localization capability—it’s always outsourced.

Manual mask annotation + I2V synthesis + VLM filtering
    |
    v
Expensive, slow, error-prone data pipeline
    |
    v
Limited task diversity in video editing (vs. image editing)
    |
    v
Assumption: image edit pairs carry enough structure
            to generate video edits via pixel-pair warping
    |
    v
Pixel-pair temporal warped flow field
    |
    v
Real-time video edit data from image edits
    +
Modality mimic loss aligns I/V output distributions
    +
Sense-related tasks internalize localization
    |
    v
Single model: unified image + video editing
               with no external mask/MLLM at inference

The Increment

One sentence: Before this paper, turning image editing data into video editing data required an expensive multi-stage pipeline with manual annotation; after this paper, a pixel-pair warped flow field does it in real time from raw image edit pairs, and a unified model learns both modalities without external localization aids.

Core Mechanism

FlowMimic has three interlocking components that tackle the three gaps above.

Component 1: Pixel-Pair Temporal Warped Flow Field. Given an image editing pair (original image, edited image), the method computes a dense correspondence—a “flow field”—that maps each pixel in the original to its counterpart in the edited version. This is a pixel-pair relationship: for every location, you know where that content moved or changed. To generate a video editing sample, the flow field is warped through time: a temporal parameter interpolates the transformation, producing a smooth sequence of frames that transition from the original image to the edited version. The key insight is that this warping happens in real time during training, so you don’t need to pre-generate and store massive video datasets. The image edit pair is the seed; the flow field is the generative engine.

Component 2: Modality Mimic Loss. The authors treat an image as a video with zero motion—a single-frame special case. To prevent the model from developing split personalities (good at images, mediocre at video or vice versa), they introduce two losses: a modality mimic generation loss and a modality mimic editing loss. These losses work by mutual imitation: the image branch tries to match the output distribution of the video branch, and vice versa. This bidirectional alignment keeps the two modalities’ capabilities in sync, so improving one doesn’t degrade the other.

Component 3: Internalized Localization via Sense-Related Tasks. Rather than requiring an external mask or MLLM at inference, the model learns to localize editing regions on its own. The authors add auxiliary tasks—most notably referring expression segmentation (RES)—that force the model to ground language instructions in spatial regions. They pair this with two novel losses: an editing-region-aware latent-level loss that supervises the latent representation to be spatially precise, and an attention-level loss that shapes the cross-attention maps to focus on the correct regions. Together, these make localization an emergent capability of the model itself.

Image Edit Pair (orig, edited)
        |
        v
+----------------------------+
| Pixel-Pair Flow Field      |
| (dense pixel correspond.)  |
+----------------------------+
        |
        | temporal warp
        v
+----------------------------+
| Generated Video Edit Seq.  |
+----------------------------+
        |
   +----+--------+
   |             |
   v             v
+----------+  +-------------------+
| Modality |  | Sense-Related     |
| Mimic    |  | Tasks (RES, etc.) |
| Losses   |  +-------------------+
+----------+        |
   |          +-----+------+
   v          |            |
I/V dist.   Latent-level  Attention-level
alignment   region loss   region loss
   |          |            |
   v          v            v
Unified    Spatially     Self-grounded
output     precise       instruction
           latents       localization

Structural Metaphor: The Master Potter’s Apprentice.

Imagine a master potter (the image editing model) who can shape any single piece beautifully—a vase, a bowl, a plate. Now you need to produce a coordinated set for a dinner service (video editing: a sequence of frames that must be consistent across time). The traditional approach is to hire a separate apprentice (I2V model), teach them to mimic the master’s style using reference pieces (ControlNet guidance), then have a critic (VLM) throw away the bad pieces. FlowMimic instead teaches the apprentice a trick: take the master’s before and after pieces (the image edit pair) and study the exact deformation pattern (the pixel-pair flow field). By replaying that deformation smoothly over time (temporal warp), the apprentice can produce the entire dinner service in real time—no separate training set of dinner services needed. The modality mimic loss is like making the master and apprentice share a workshop and critique each other’s work, so neither develops bad habits in isolation. And the sense-related tasks are like teaching the apprentice to read a customer’s verbal order (“make the handle thinner”) and know exactly which part of the clay to reshape, without the customer pointing at it. Without this metaphor, you’d be staring at three separate loss functions and wondering why they’re all in the same paper. With it, you see: the flow field converts data, the mimic loss aligns quality, and the sense tasks teach grounding.

Key Concepts

• Pixel-Pair Warped Flow Field: Think of it like this. You have two photos: your living room, and your living room with a red couch instead of a blue one. For every pixel on the blue couch, there’s a corresponding pixel on the red couch at roughly the same position. This pixel-to-pixel mapping *is the flow field. Now imagine replaying that transformation smoothly across 16 frames—blue gradually morphing into red, with realistic texture changes. That’s the “temporal warp.” The genius is that this requires no video training data at all; it’s computed on the fly from the image pair. It’s the difference between filming a real couch transformation (expensive, impractical) and mathematically simulating what the transformation would look like if it happened over time (cheap, automatic).

• Modality Mimic Loss: Most multi-modal models have a subtle problem: the image branch and video branch drift apart during training, each optimizing for its own domain. Modality mimic loss is a forcing function that says “your outputs should look like mine, and mine should look like yours.” It’s bidirectional distillation. Concretely, when generating an image, the model is penalized if its output distribution doesn’t resemble what it would produce for a single-frame video. And vice versa. This keeps the two branches from diverging, so you get one model with two coherent capabilities rather than two half-models stitched together.

• Internalized Localization: Existing video editing models play a frustrating game of telephone: the user says “change the sky to sunset,” an MLM translates that into a mask or bounding box, and the editing model uses that mask as a crutch. FlowMimic cuts out the middleman. By training on referring expression segmentation (“segment the object described as ‘the red car on the left’”), the model learns to internally map language to spatial regions. The latent-level loss ensures the internal representation is spatially sharp; the attention-level loss ensures the cross-attention mechanism actually attends to the right area. After training, you just say “change the sky” and the model knows where the sky is—no mask needed.

Framework Shift

Before (mainstream approach):
+----------+     +----------+     +---------+
| Manual   |---->| I2V +    |---->| VLM     |
| Mask     |     | ControlN.|     | Filter  |
| Annot.   |     | Synth.   |     | /Refine |
+----------+     +----------+     +---------+
        |              |              |
        v              v              v
+------------------------------------------+
| Limited, expensive video editing dataset  |
+------------------------------------------+
                  |
                  v
+------------------------------------------+
| Video-only model + external MLLM / masks |
+------------------------------------------+

After (this paper):
+------------------+
| Image Edit Pairs |  (abundant, diverse)
+------------------+
        |
        v
+---------------------------+
| Pixel-Pair Warped Flow   |
| Field (real-time gen.)   |
+---------------------------+
        |
        v
+-------------------------------------------+
| Unified model (image = zero-frame video)  |
| + Modality mimic loss (I/V alignment)     |
| + Sense tasks (internalized localization) |
+-------------------------------------------+

From a multi-stage, human-in-the-loop data pipeline feeding a modality-specific model to a single-stage, automated flow field feeding a unified model with internalized localization—the core shift is replacing curated data collection with mathematical data transformation, and replacing external localization tools with learned internal grounding.

Expert Assessment

Problem choice: This is a real gap. Video editing data scarcity is a well-known bottleneck, and the field has largely brute-forced its way through it with expensive pipelines. The insight that image editing data is abundant and can be structurally converted to video editing data is genuine and worth pursuing. It sits at a natural inflection point: image editing models are maturing, video editing is the obvious next frontier, and the data wall is the first thing you hit.

Method maturity: The pixel-pair warped flow field is a clever structural insight, not brute force—it exploits the geometric relationship between edit pairs rather than just throwing more compute at the problem. The modality mimic loss is straightforward but sound. The internalized localization via sense-related tasks is the most ambitious component and likely the most fragile in practice; referring expression segmentation is a hard task, and whether auxiliary training on it reliably transfers to open-ended editing instructions is an empirical question that needs more probing. One concern: the paper bundles three distinct contributions (data generation, modality alignment, localization internalization) into one system, which makes it hard to isolate which component drives the gains.

Experimental integrity: The claim that a model can learn video editing from *only synthetically generated data is strong and testable, which is good. However, the abstract alone doesn’t reveal whether ablation studies cleanly separate the three contributions. If the paper presents only the full system’s results without systematic ablations of each component, that would be a significant weakness. Fair baselines are critical here—comparing against models that use real video editing data is tricky because the data regimes are fundamentally different.

Writing quality: The abstract is overloaded. It tries to sell three ideas at once (data generation, modality mimicry, internalized localization) and ends up doing justice to none of them in the limited space. A reader unfamiliar with the field would struggle to identify the single most important contribution. The methods section would benefit from a cleaner narrative: lead with the flow field as the hero, then show how the other components support it. As written, the paper reads like three mini-papers sharing a title page.

Verdict: weak accept — The core idea of pixel-pair warped flow for real-time video edit data generation is a genuinely useful contribution that could shift how the field thinks about data. The other two components are interesting but underdeveloped, and the paper’s ambition may dilute its impact. With stronger ablations and a tighter narrative, this could be a solid accept.

Takeaways

Three specific ideas worth stealing:

1. Structural data conversion over data collection. The paradigm of “don’t collect more data, mathematically transform the data you have” is broadly applicable. If you have paired data in one modality (e.g., before/after images), flow-based warping can bootstrap sequences in another modality (video). This pattern could apply to audio (morphing between audio clips), 3D (interpolating between 3D edits), or even text (interpolating between paraphrases for data augmentation).

2. Modality-as-special-case framing. Treating images as zero-frame videos, rather than as a separate modality, is a design choice that simplifies architecture and training. Any multi-modal system where one modality is a degenerate case of another (2D as flat 3D, static as zero-velocity dynamic) can borrow this framing to unify training.

3. Localization as an auxiliary training objective, not an inference-time dependency. Instead of bolting on an external tool for spatial grounding, add segmentation-style tasks during training and shape the latent space and attention maps to internalize it. This “bake it in, don’t bolt it on” approach could save inference-time complexity in many conditional generation systems.