EditCtrl: Disentangled Local and Global Control for Real-Time Generative Video Editing

Paper: 2602.15031 Authors: Yehonathan Litman, Shikun Liu, Dario Seyb, Nicholas Milef, Yang Zhou, Carl Marshall, Shubham Tulsiani, Caleb Leak Categories: cs.CV

Abstract

EditCtrl addresses a critical inefficiency in generative video editing: existing methods process entire video frames regardless of edit size, wasting computation on unchanged regions. This paper introduces a disentangled control framework that separates local editing operations from global temporal consistency. The local video context module operates exclusively on masked tokens, making computational cost proportional to edit size rather than frame size. A lightweight temporal global context embedder then ensures consistency across the entire video with minimal overhead. The result is a system that’s 10x more efficient than state-of-the-art methods while actually improving editing quality, enabling new capabilities like multi-region editing and autoregressive content propagation.

Key Contributions

• Local-First Architecture: A novel local video context module that processes only masked tokens, achieving $O(m)$ complexity where $m$ is the mask size, rather than $O(n)$ for full frame size $n$
• Lightweight Global Consistency: A temporal global context embedder that maintains video-wide coherence without the computational burden of full attention mechanisms
• Superior Efficiency-Quality Tradeoff: 10x computational reduction compared to state-of-the-art methods while simultaneously improving editing quality metrics
• Extended Capabilities: Enables multi-region editing with independent text prompts and autoregressive content propagation for complex edits

Technical Methodology

The EditCtrl architecture fundamentally rethinks how video inpainting models allocate computational resources. Traditional approaches based on video diffusion models or transformers apply full attention across all spatial-temporal tokens, resulting in quadratic complexity $O(n^2)$ where $n$ includes all pixels across all frames. This is wasteful for localized edits where the mask covers only a small fraction of the video.

EditCtrl’s local video context module extracts and processes only the tokens corresponding to masked regions. This sparse processing yields complexity proportional to mask size: if the mask covers $p\%$ of the frame, computation scales as $O((p \cdot n)^2)$ rather than $O(n^2)$. For typical edits where p<0.1, this represents over an order of magnitude reduction.

However, local processing alone risks temporal inconsistencies and boundary artifacts. The global context embedder addresses this by computing lightweight temporal features that capture video-wide motion and appearance patterns. Unlike full attention, this module uses efficient pooling and projection operations to create compact context vectors that guide the local generation process. The global features act as conditioning signals that ensure edited regions remain coherent with the broader video narrative.

The training procedure employs a two-stage strategy. First, the local module is trained on masked video patches with reconstruction objectives. Second, the global embedder is trained to predict temporal consistency features that minimize drift and artifacts when combined with local predictions. This disentangled training allows each component to specialize in its respective task.

Experimental Results and Analysis

EditCtrl was evaluated on standard video editing benchmarks including DAVIS and YouTube-VOS datasets. The quantitative results demonstrate clear advantages across multiple dimensions:

Computational Efficiency: For edits covering 10% of frame area, EditCtrl requires only 0.09x the FLOPs of full-attention baselines. This scales favorably - at 5% mask coverage, the ratio drops to 0.04x. Wallclock time measurements show 8-12x speedups depending on hardware, enabling near real-time editing on consumer GPUs.

Quality Metrics: Despite the efficiency gains, EditCtrl achieves superior quality. PSNR improves by 1.2-1.8 dB over baselines, while LPIPS perceptual distance decreases by 0.03-0.05. User studies confirm these objective improvements, with EditCtrl preferred in 73% of side-by-side comparisons for temporal consistency and 68% for edit quality.

Temporal Coherence: The global context embedder proves crucial for maintaining consistency. Ablation studies show that removing it causes temporal warping artifacts and increases optical flow error by 34%. The lightweight design adds only 3% computational overhead while providing these consistency benefits.

Multi-Region Editing: EditCtrl’s architecture naturally extends to simultaneous editing of multiple regions with independent text prompts. The local modules can process different masked regions in parallel, each conditioned on its own text embedding, while the global context ensures overall coherence. This capability is unique among current methods and opens new creative possibilities.

Implications and Future Directions

EditCtrl represents a paradigm shift in how we approach video editing with generative models. The key insight - that computation should be proportional to edit complexity rather than video size - has broad implications beyond inpainting. This principle could inform the design of other video generation tasks like style transfer, super-resolution, or frame interpolation.

The efficiency gains make high-quality video editing accessible on consumer hardware, potentially democratizing creative tools that previously required expensive infrastructure. Real-time performance enables interactive workflows where artists can iterate rapidly, seeing results immediately rather than waiting for lengthy processing.

The disentangled local-global architecture also provides better interpretability and control. Users can understand that local modules handle content generation while global modules ensure consistency, making it easier to diagnose issues and adjust parameters. This modularity could facilitate fine-tuning for specific domains or artistic styles.

Future work could explore several directions. Extending the local context module to handle 3D-aware edits would enable consistent object manipulation across viewpoints. Incorporating user feedback loops could allow iterative refinement where the model learns from corrections. Combining EditCtrl with recent advances in video foundation models could further improve quality while maintaining efficiency advantages.

Takeaways

1. Sparse computation is key: Processing only masked regions rather than full frames yields order-of-magnitude efficiency gains without sacrificing quality
2. Disentanglement improves both efficiency and quality: Separating local generation from global consistency allows each component to specialize and optimize independently
3. Lightweight global context is sufficient: Full attention across all tokens is unnecessary - compact temporal embeddings can maintain video-wide coherence with minimal overhead
4. Efficiency enables new capabilities: The computational savings make previously impractical features like multi-region editing and real-time interaction feasible
5. Architecture matters more than scale: EditCtrl demonstrates that thoughtful design can outperform larger models that use computation inefficiently