Aurora: Bridging User Intent and Video Editing Models with Tool-Using Agents

Paper: 2605.18748 Authors: Yongsheng Yu, Ziyun Zeng, Zhiyuan Xiao, Zhenghong Zhou, Hang Hua, Wei Xiong, Jiebo Luo Categories: cs.CV

The Gap

Recent unified video editing models (like Gen-3, CogVideoX-Fun) converge on a single design: one diffusion transformer jointly consumes text, source video, and reference images to handle replacement, removal, style transfer, and insertion. The architecture is flexible, but it assumes users arrive with model-ready inputs—precise text prompts, curated reference images, and spatial masks for local edits. Real user requests rarely provide these. A user might say “make it look like a Miyazaki film” without supplying reference frames, or “remove the car” without specifying which car or where. The model has no mechanism to resolve this underspecification, so it either fails or produces arbitrary results.

Prior work treats this as a prompt engineering problem: better instructions yield better outputs. But instruction tuning alone cannot bridge the gap when the user fundamentally lacks the visual or spatial information the model requires. The missing piece is a reasoning layer that can interpret vague intent, retrieve missing visual references, and ground spatial edits before generation begins.

Problem: User says "make it Miyazaki-style"
   |
   v
Assumption: Model needs reference images + precise text
   |
   v
Method: VLM agent plans edit, retrieves references, refines prompt
   |
   v
Evidence: AgentEdit-Bench shows 15-20% improvement over instruction-only
   |
   v
Conclusion: Agentic planning resolves underspecification before generation

The Increment

One sentence: Before Aurora, unified video editing models required users to provide model-ready conditioning inputs; after Aurora, a VLM agent translates raw, underspecified requests into structured edit plans with retrieved references and spatial grounding.

Core Mechanism

Aurora pairs a tool-augmented vision-language model agent with a unified video diffusion transformer. The VLM agent receives a raw user request and the source video, then produces a structured edit plan that specifies: (1) refined text prompts aligned with the transformer’s conditioning channels, (2) reference images retrieved from external databases or extracted from the video itself, and (3) spatial masks for local edits. The agent has access to tools for image search, frame extraction, mask generation, and instruction refinement. Once the plan is complete, the diffusion transformer consumes the structured inputs and generates the edited video.

The agent is trained in two stages. First, supervised fine-tuning on complete edit planning tasks teaches it to map underspecified requests to structured plans. The training data includes examples where the agent must retrieve reference images for style transfer, extract frames for object insertion, and generate masks for local removal. Second, preference optimization using pairs of successful and failed tool invocations teaches the agent robust tool use and instruction refinement. The preference pairs penalize hallucinated tool calls, redundant retrievals, and vague prompts.

The diffusion transformer itself is a standard unified architecture—text, video, and reference images are tokenized and fed into a joint attention mechanism. Aurora does not modify the transformer’s weights or architecture. Instead, it ensures that the transformer receives well-formed inputs by delegating the reasoning and retrieval work to the VLM agent upstream.

User Request: "make it look like a Miyazaki film"
   |
   v
VLM Agent:
   [Tool: Image Search] -> retrieves Miyazaki reference frames
   [Tool: Instruction Refine] -> "apply hand-drawn animation style
                                  with soft colors and pastoral scenery"
   |
   v
Structured Plan:
   - Text: refined prompt
   - Reference: retrieved images
   - Mask: none (global edit)
   |
   v
Diffusion Transformer -> Edited Video

Think of Aurora as a restaurant with a sommelier. The diffusion transformer is the kitchen—it can cook anything, but only if you give it precise ingredients and instructions. Most customers walk in saying “I want something Italian” without specifying pasta type, sauce, or wine pairing. The sommelier (VLM agent) listens to the vague request, asks clarifying questions (tool calls), retrieves the right wine from the cellar (reference images), and translates the customer’s intent into a detailed order the kitchen can execute. The kitchen’s skill hasn’t changed, but the sommelier ensures it receives actionable instructions instead of guesswork.

Key Concepts

• Underspecification: A user request is underspecified when it omits information the model requires to generate a coherent output. For video editing, this typically means missing reference images (style transfer without examples), missing spatial grounding (local edits without masks), or vague text (ambiguous object descriptions). Underspecification is not a failure of the user—it reflects the natural way people communicate intent. A user might say “make the sky more dramatic” without knowing they need to provide a reference image of a dramatic sky or specify which part of the frame contains the sky. The model cannot resolve this ambiguity alone because it has no mechanism to retrieve external knowledge or infer spatial regions from text. Aurora addresses underspecification by inserting a reasoning layer (the VLM agent) that can interpret vague intent, retrieve missing information, and ground spatial edits before generation.

• Tool-augmented VLM agent: A vision-language model equipped with external tools that extend its capabilities beyond text generation. In Aurora, the agent has access to image search (retrieve reference images from databases), frame extraction (sample frames from the source video), mask generation (produce spatial masks for local edits), and instruction refinement (rewrite vague prompts into precise conditioning text). The agent decides which tools to invoke based on the user request and the current state of the edit plan. For example, if the user asks for style transfer without providing references, the agent calls the image search tool. If the user requests local removal without specifying a region, the agent calls the mask generation tool. Tool use is learned through supervised fine-tuning on complete edit planning tasks and preference optimization on tool invocation pairs.

• Structured edit plan: The intermediate representation that bridges the user request and the diffusion transformer’s conditioning channels. A structured plan specifies: (1) refined text prompts that align with the transformer’s text encoder, (2) reference images for style transfer or object insertion, and (3) spatial masks for local edits. The plan is model-ready—it can be directly consumed by the diffusion transformer without further interpretation. The VLM agent’s job is to map the raw user request to this structured representation by resolving underspecification through tool use and reasoning. The plan format is fixed and known to the agent during training, so the agent learns to produce plans that match the transformer’s expected input schema.

Framework Shift

Before (instruction-only):              After (Aurora):

User Request                            User Request
     |                                       |
     v                                       v
"make it Miyazaki-style"               VLM Agent
     |                                  /    |    \
     v                                 /     |     \
Diffusion Transformer            Search  Extract  Refine
     |                                 \     |     /
     v                                  \    |    /
Arbitrary output                    Structured Plan
(no references)                          |
                                         v
                                  Diffusion Transformer
                                         |
                                         v
                                  Coherent output
                                  (with references)

From direct instruction-to-generation to agent-mediated planning, the core shift is inserting a reasoning layer that resolves underspecification before the model sees the request.

Expert Assessment

Problem choice: This is a real gap. Unified video editing models have converged on a flexible conditioning design, but the flexibility is wasted if users cannot provide the required inputs. The paper correctly identifies that instruction tuning alone cannot solve this—you need a reasoning layer that can retrieve and ground missing information. The problem sits at the intersection of video generation and agentic AI, which is timely given the recent push toward tool-augmented LLMs.

Method maturity: The method is straightforward—pair a VLM agent with a frozen diffusion transformer and train the agent to produce structured plans. The insight is not particularly deep, but it is practical. The agent training pipeline (supervised fine-tuning + preference optimization) is borrowed from standard LLM agent work, and the diffusion transformer is off-the-shelf. The contribution is in the integration and the task formulation, not in novel algorithmic components. A simpler approach might be to fine-tune the diffusion transformer itself to handle underspecified inputs, but that would require retraining the model for each new type of underspecification, whereas the agent approach is more modular.

Experimental integrity: The paper introduces AgentEdit-Bench, a new benchmark for evaluating agent-enhanced video editing under textual and visual underspecification. The benchmark design is reasonable—it includes tasks where reference images are missing, spatial grounding is absent, or text prompts are vague. The baselines are fair: instruction-only models (no agent), retrieval-augmented models (agent without planning), and ablations of the agent’s tool set. The numbers show consistent 15-20% improvement over instruction-only baselines across multiple metrics (CLIP similarity, user preference, edit fidelity). One red flag: the paper does not report failure modes or cases where the agent’s tool calls are incorrect. The preference optimization should reduce hallucinated tool calls, but the paper does not quantify how often this still happens.

Writing quality: The paper is well-structured and easy to follow. The introduction clearly motivates the problem, and the method section is detailed enough to reproduce. The weakest section is the related work—it lumps together video editing models, VLM agents, and tool-augmented LLMs without drawing clear distinctions. A rewrite that explicitly contrasts Aurora’s agent-mediated planning with prior retrieval-augmented generation approaches would elevate the paper. The experimental section is thorough, but the ablation studies could be more granular (e.g., which tools contribute most to performance?).

Verdict: weak accept — The problem is real, the method is practical, and the experiments are solid, but the novelty is incremental. The contribution is in the integration and task formulation, not in new algorithmic insights. The paper would be stronger with more analysis of failure modes and a deeper ablation of the agent’s tool use.

Takeaways

Practitioners building multimodal generation systems can steal the agent-as-planner pattern: instead of expecting users to provide model-ready inputs, insert a reasoning layer that translates underspecified requests into structured plans. The key is to define a fixed plan schema that matches the model’s conditioning channels, then train the agent to produce plans in that schema using supervised fine-tuning and preference optimization. This pattern transfers beyond video editing—any generation model that requires structured inputs (image editing, 3D synthesis, audio generation) can benefit from an upstream agent that resolves underspecification through tool use and reasoning.

The preference optimization for tool use is also worth stealing. Instead of only training the agent on successful tool invocations, collect pairs of successful and failed invocations (e.g., correct vs. hallucinated tool calls, precise vs. vague prompts) and use preference optimization to teach the agent robust tool use. This is more sample-efficient than pure supervised learning and helps the agent generalize to novel tool combinations.

Finally, the modular design (frozen diffusion transformer + trainable agent) is practical for real-world deployment. You can swap out the diffusion transformer for a newer model without retraining the agent, as long as the conditioning schema remains compatible. This modularity reduces the cost of keeping the system up-to-date as video generation models improve.