GraphVid: Controlling Video Generation with Scene Graphs, Not Scribbles

Paper: 2607.21580 Authors: Vedant Shah, Onkar Susladkar, Tushar Prakash, Kiet Nguyen, Tianjio Yu, Adheesh Juvekar, Muntasir Waheed, Ismini Lourentzou Categories: cs.CV, cs.AI

The Gap

Existing controllable video generation research has hit a wall with input specification. The dominant methods are either text-only (ambiguous for complex interactions) or trajectory-based (e.g., drag-to-move). The latter requires users to manually draw precise motion paths for *every single object. This approach crumbles under scene complexity: it doesn’t scale, becomes ambiguous during occlusions, and tells the model where things move pixel-by-pixel, but not what relationship they have (e.g., “A pushes B,” “C follows D”). This is a fundamental interface limitation, not just a model capacity issue.

The logical path from gap to conclusion is:

Problem: Existing control interfaces (text, trajectories) are imprecise or unscalable for multi-object interaction.
     |
     v
Assumption: A structured, semantic representation of object relationships can provide better control.
     |
     v
Method: Introduce GraphVid, which uses a Scene Graph as a conditional interface. Also, create a new dataset (GraphVid-Bench) with structured annotations.
     |
     v
Evidence: GraphVid outperforms a strong trajectory-based baseline (Motion-I2V) on FID, FVD, PSNR, SSIM while using less data and fewer parameters.
     |
     v
Conclusion: Structured semantic interfaces (like scene graphs) are a powerful, underexplored paradigm for controllable video generation.

The Increment

One sentence: Before this paper, specifying precise multi-object interactions in video generation required tedious, pixel-level trajectory drawing; after, users can define high-level relational graphs (e.g., “person_A high-fives person_B”) and let the model handle the complex motion synthesis.

Core Mechanism

GraphVid is an image-to-video diffusion model conditioned on a scene graph. The pipeline has three key stages: 1) Graph Encoding, 2) Cross-Attention Fusion, and 3) Video Decoding. First, the user provides an initial image and a scene graph specifying objects and their relationships (e.g., (Object_A, interacts_with, Object_B)). This graph is processed by a Scene Graph Transformer into rich node and edge embeddings. These embeddings are not just labels; they encode spatial and semantic context about the relationships.

During the video denoising process (in the diffusion U-Net), these graph embeddings are injected via cross-attention mechanisms. This means that at every denoising step, the model “looks at” the graph embeddings to understand which objects are interacting and how. This is fundamentally different from a single text prompt; it provides a structured, disentangled signal for each relationship in the scene. The model then synthesizes the video frames conditioned on this graph-aware signal and the initial image, learning to translate semantic relations into plausible motions.

[User Input]---->[Scene Graph + Image]
                       |
                       v
             [Scene Graph Encoder]
                 (Node/Edge Embeddings)
                       |
                       v
[Diffusion U-Net]<----| (Cross-Attention Injection at multiple scales)
      ^                |
      |                v
 [Noisy Video] [Graph-Conditioned Denoising]
      |                |
      v                v
[Clean Video]<----[Output Video]

Structural Metaphor: Think of it like a play director with a script and actors.

• The initial image is the stage set.
• The scene graph is the director’s script. It doesn’t specify every step (trajectory), but defines character roles (nodes) and their key interactions (“fight scene with,” “hand object to” - edges).
• The Scene Graph Encoder is the director interpreting the script, understanding character motivations and dynamics.
• The Cross-Attention mechanism in the U-Net is the director giving real-time instructions to the actors (pixels) during filming. At each moment (denoising step), the director checks the script to ensure the actors are performing the right relationship.
• The final video is the filmed scene that follows the director’s high-level instructions, with actors moving naturally and interactively. You don’t need to choreograph every footstep; you define the drama, and the system choreographs it.

Key Concepts

• Scene Graph as Control Interface: Forget pixels or text. A scene graph is a small network diagram where circles (nodes) are objects (e.g., “car,” “person”) and arrows (edges) are labeled relationships (“behind,” “holding,” “collision_course”). This is the core idea: the user’s intent is expressed not as a single sentence or a squiggle, but as a structured map of *who is doing what to whom. This directly encodes the multi-object interactions that were previously ambiguous. Example: For a scene of a robot handing a tool to a human, the graph is: [robot] --(hands_tool_to)--> [human], which is far clearer than describing the motion paths of both the robot arm and the human hand.

• Graph Relation Propagation: This is the technical heart. The model doesn’t just see the graph; it learns how the *meaning of an edge (the relationship) influences the motion of the connected nodes over time. During training, the model sees many examples where a “push” edge leads to specific joint motion patterns between objects. During inference, it propagates this understanding through the cross-attention layers, so that when it’s generating a frame, it “remembers” that object A is supposed to be “pushing” object B, synthesizing appropriate velocities and deformations for both.

Framework Shift

Before (mainstream approach):        After (this paper):
User: Draw precise motion paths      User: Draw a semantic relationship graph
for each object.                     (Object A -> Relation -> Object B).
     |                                    |
     v                                    v
Control Signal: Pixel-level          Control Signal: High-level, structured
trajectories (per-object).           object relationships.
     |                                    |
     v                                    v
Model: Learns to move pixels         Model: Learns to synthesize interactions
along given paths. Ambiguous         that fulfill graph semantics. More
under occlusion. Poor scaling.       scalable to complex scenes.

From pixel-motion specification to semantic-relation specification, the core shift is *moving the control interface from the how (motion trajectory) to the what (object relationship).

Expert Assessment

Problem choice: Excellent. This is a real and underappreciated gap. The field has focused heavily on model architecture and training objectives, but the *human-AI interface for control is often an afterthought. Fixating on pixel trajectories is a dead end for usability. This paper correctly identifies semantic structure as a better interface and tackles it head-on.

Method maturity: A clever, well-motivated integration of known parts (Scene Graph Transformers, diffusion cross-attention) rather than a brute-force new architecture. The insight is in the framing and the interface design. However, the method likely inherits limitations from scene graphs themselves—they can be cumbersome to define for highly dynamic or fluid scenes (e.g., a flock of birds). A simpler approach might have been overlooked: can the model learn to extract or infer such a graph from video data in a self-supervised way, reducing user burden?

Experimental integrity: The numbers are compelling, especially the PSNR/SSIM jumps, which suggest more precise control. Reducing FID/FVD while improving controllability is a strong signal. Red flags: 1) The comparison is primarily against one strong baseline (Motion-I2V). How does it fare against other recent non-trajectory methods? 2) GraphVid-Bench is curated by them; while they describe the process, this is a common point of bias. The results need validation on broader benchmarks. 3) The “less data/parameters” claim is great for efficiency but needs context—is the baseline just bloated?

Writing quality: Clear and well-structured. The abstract and intro effectively sell the story. A section that could elevate it: A deeper failure case analysis. When does the graph condition break down? What kinds of relationships are hard to model? Showing limitations would bolster credibility more than just reporting metrics. The related work could also better position this within the broader “controllable generation” taxonomy beyond just video.

Verdict: Weak Accept — It presents a compelling and clean idea (graph-as-interface) that fills a genuine gap, with solid initial results. The main weakness is the scope of evaluation (single main baseline, proprietary benchmark). It’s a strong conceptual contribution that should spark follow-up work.

Takeaways

1. Steal the Interface Design Philosophy: The most transferable idea is: when designing a generative control system, think about the *structure of human intent. Can you move from a flat signal (text, points) to a structured representation (graphs, trees, programs) that better mirrors how users conceptualize the problem? This applies to image editing, audio synthesis, and 3D generation.
2. Small Data, Smart Conditioning: GraphVid achieves strong results with less data. The takeaway is that a well-chosen, semantically rich conditional signal (the graph) can be more data-efficient than a weak, ambiguous signal (pixel trajectories) paired with more data. For practitioners, investing in a better conditional input design might yield better returns than simply scaling up data and compute.
3. The “Director” Pattern: For any multi-agent or multi-object generation task, consider the “director” pattern: separate the high-level script/relations from the low-level execution. Use an encoder to interpret the script and a mechanism (like cross-attention) to enforce it during generation. This modularity can make systems more controllable and debuggable.