FlexAM: Flexible Appearance-Motion Decomposition for Versatile Video Generation Control

Paper: 2602.13185 Authors: Mingzhi Sheng, Zekai Gu, Peng Li, Cheng Lin, Hao-Xiang Guo, Ying-Cong Chen, Yuan Liu Categories: cs.CV, cs.GR

Abstract

FlexAM addresses the fundamental challenge of controllable video generation by proposing a novel approach to disentangle appearance and motion. Unlike existing methods that rely on ambiguous or task-specific control signals, FlexAM introduces a unified framework built on 3D point cloud representations of video dynamics. The framework incorporates multi-frequency positional encoding for fine-grained motion distinction, depth-aware positional encoding for spatial understanding, and flexible control signals that balance generation precision with quality. This architecture enables FlexAM to handle diverse tasks including image-to-video (I2V) generation, video-to-video (V2V) editing, camera control, and spatial object manipulation, demonstrating superior performance across all evaluated benchmarks.

Key Contributions

• 3D Point Cloud Control Signal: Introduces a novel representation of video dynamics as point clouds, providing a more structured and interpretable control mechanism compared to traditional 2D or latent-based approaches
• Multi-Frequency Positional Encoding: Implements hierarchical encoding schemes that capture both coarse and fine-grained motion patterns, enabling precise control over temporal dynamics
• Depth-Aware Encoding: Incorporates spatial depth information into the control signal, allowing for better 3D scene understanding and more realistic motion generation
• Flexible Control Mechanism: Develops an adaptive control signal that can be adjusted to balance between generation precision and creative freedom, accommodating different use cases
• Unified Framework: Demonstrates that appearance-motion disentanglement enables a single model to handle multiple video generation tasks without task-specific modifications

Technical Methodology

The core innovation of FlexAM lies in its representation of video content through explicit appearance-motion decomposition. The framework operates on the principle that video can be understood as the combination of static visual elements (appearance) and their temporal evolution (motion).

Point Cloud Representation: FlexAM converts video frames into 3D point clouds where each point $\mathbf{p}_i = (x_i, y_i, z_i, t_i)$ encodes spatial coordinates and temporal information. This representation naturally captures the geometric structure of scenes and their evolution over time.

Multi-Frequency Encoding: The positional encoding scheme applies multiple frequency bands to capture motion at different scales:

$\text{PE}(\mathbf{p}) = [\sin(2^0\pi\mathbf{p}), \cos(2^0\pi\mathbf{p}), ..., \sin(2^L\pi\mathbf{p}), \cos(2^L\pi\mathbf{p})]$

where $L$ determines the number of frequency bands. This allows the model to distinguish between subtle movements and large-scale motion patterns.

Depth-Aware Processing: The framework integrates depth information $d_i$ for each point, enabling the model to understand occlusions, relative positioning, and 3D spatial relationships. This is particularly crucial for camera control tasks where maintaining geometric consistency is essential.

Control Signal Flexibility: FlexAM introduces a learnable weighting mechanism that modulates the influence of the control signal, allowing users to trade off between strict adherence to control inputs and generative diversity.

Experimental Results and Performance

FlexAM demonstrates state-of-the-art performance across multiple video generation benchmarks. The experiments validate the framework’s versatility and effectiveness in handling diverse control scenarios.

Image-to-Video Generation: On I2V tasks, FlexAM achieves superior temporal consistency while maintaining high visual fidelity to the input image. The appearance-motion decomposition allows the model to preserve fine details from the source image while generating natural motion patterns.

Video Editing: For V2V editing tasks, FlexAM enables precise modifications to either appearance or motion independently. Users can change object appearances while maintaining original motion, or alter motion patterns while preserving visual characteristics.

Camera Control: The depth-aware encoding proves particularly effective for camera control tasks. FlexAM accurately interprets camera trajectory specifications and generates videos with correct perspective changes, parallax effects, and geometric consistency.

Spatial Object Editing: The point cloud representation facilitates intuitive spatial manipulation. Users can move, scale, or modify objects in 3D space, and FlexAM generates corresponding video sequences that respect physical constraints and maintain temporal coherence.

Quantitative Metrics: Across benchmarks, FlexAM shows improvements in Fréchet Video Distance (FVD), temporal consistency scores, and user preference studies compared to existing methods. The flexible control mechanism also demonstrates better balance between control accuracy and generation quality.

Implications and Future Directions

FlexAM represents a significant step toward more controllable and versatile video generation systems. The appearance-motion decomposition paradigm offers several advantages over existing approaches:

Interpretability: The explicit separation of appearance and motion provides users with intuitive control mechanisms that align with human understanding of video content.

Generalization: By learning fundamental decomposition rather than task-specific patterns, FlexAM generalizes better to novel scenarios and control requirements.

Composability: The framework’s modular design allows for combining different control signals, enabling complex editing operations that would be difficult with monolithic approaches.

Scalability: The point cloud representation scales naturally to higher resolutions and longer videos, as it focuses on structural information rather than pixel-level details.

Future research directions include extending the framework to handle more complex scene dynamics, incorporating physical constraints for realistic motion generation, and exploring applications in interactive video creation tools. The appearance-motion decomposition principle could also inform other generative modeling domains beyond video.

Takeaways

1. Fundamental Decomposition: Separating appearance and motion provides a more robust foundation for video generation control than task-specific or ambiguous signals
2. 3D Representation Advantage: Point cloud representations of video dynamics enable more structured and interpretable control mechanisms
3. Multi-Scale Motion Capture: Multi-frequency positional encoding is essential for capturing both fine-grained and coarse motion patterns
4. Unified Framework Benefits: A single model based on appearance-motion disentanglement can handle diverse video generation tasks without architectural modifications
5. Flexibility-Quality Tradeoff: Adaptive control signals allow users to balance between precise control and generative freedom based on application requirements
6. Depth Integration: Incorporating depth information significantly improves performance on tasks requiring 3D spatial understanding, particularly camera control
7. Generalization Capability: Learning fundamental decomposition principles leads to better generalization across different video generation scenarios compared to task-specific approaches