VideoSketcher: Leveraging Video Diffusion Models for Sequential Sketch Generation

Paper: 2602.15819 Authors: Hui Ren, Yuval Alaluf, Omer Bar Tal, Alexander Schwing, Antonio Torralba, Yael Vinker Categories: cs.CV

Abstract

VideoSketcher addresses a fundamental limitation in sketch generation: most models treat sketches as static images, ignoring the sequential nature of drawing. This work presents a novel approach that adapts pretrained text-to-video diffusion models to generate sketching processes as temporal sequences. The key innovation lies in combining large language models for semantic planning and stroke ordering with video diffusion models for high-quality visual rendering. By representing sketches as short videos showing progressive stroke formation on a blank canvas, the method achieves remarkable data efficiency—requiring only seven manually authored sketching processes for training while producing high-quality, temporally coherent sequential sketches that follow text-specified ordering instructions.

Key Contributions

• Sequential sketch representation: Frames sketching as a video generation problem where strokes progressively appear on canvas, capturing the temporal structure of drawing
• Hybrid architecture: Combines LLMs for semantic stroke ordering with video diffusion models for visual rendering, leveraging complementary strengths
• Two-stage fine-tuning strategy: Decouples stroke ordering learning (using synthetic data) from appearance learning (using minimal human-drawn data)
• Extreme data efficiency: Achieves high-quality results with only seven manually authored sketching processes for visual appearance distillation
• Flexible extensions: Demonstrates brush style conditioning and autoregressive generation for enhanced controllability

Methodology and Technical Approach

The VideoSketcher framework operates on a fundamental insight: sketching is inherently temporal, and video models are naturally suited to capture this temporal structure. The approach consists of several key components:

Sketch-as-Video Representation: Rather than generating static images, sketches are represented as short video sequences where each frame shows the progressive addition of strokes on a blank canvas. This representation naturally encodes both the ordering of strokes and the continuous formation of individual marks.

Two-Stage Fine-Tuning: The training strategy decouples two distinct aspects of sketch generation:

1. Stroke Ordering Stage: Uses synthetic shape compositions with controlled temporal structure to teach the model how to order strokes semantically. This stage leverages programmatically generated data where shapes are composed in specific orders (e.g., “draw the circle, then the square”).

2. Appearance Distillation Stage: Fine-tunes on a minimal set of seven human-drawn sketching processes that capture authentic drawing dynamics, including both global stroke ordering and the continuous formation of individual strokes.

LLM-Guided Ordering: Large language models provide semantic understanding for stroke ordering through text prompts. For example, prompts like “draw the head first, then the body, then the limbs” guide the temporal structure of generation.

Video Diffusion Backbone: Pretrained text-to-video diffusion models serve as the rendering engine, providing strong priors for temporal coherence and visual quality without requiring extensive sketch-specific training data.

Results and Capabilities

The method demonstrates several impressive capabilities despite its minimal training data requirements:

Text-Controlled Ordering: Generated sketches faithfully follow text-specified drawing orders, enabling users to control not just what is drawn but how it unfolds temporally. This allows for pedagogical applications where specific drawing sequences can be demonstrated.

Visual Quality: Despite training on only seven human sketches, the generated sequences exhibit rich visual detail and natural stroke dynamics that closely resemble human drawing behavior.

Temporal Coherence: The video diffusion backbone ensures smooth transitions between frames, avoiding flickering or discontinuities common in frame-by-frame generation approaches.

Brush Style Conditioning: The framework extends to support different brush styles and stroke characteristics, allowing for artistic variation while maintaining temporal structure.

Autoregressive Generation: The model can generate sketches autoregressively, enabling interactive and collaborative drawing scenarios where users can guide the generation process step-by-step.

Implications and Future Directions

VideoSketcher opens several promising research directions and applications:

Data Efficiency Paradigm: The extreme data efficiency—achieving strong results with just seven training examples—suggests that pretrained video models contain powerful priors for temporal structure that can be effectively transferred to specialized domains with minimal fine-tuning.

Creative Tools: The ability to control drawing order and style makes this approach valuable for educational applications, animation, and interactive creative tools where the process of creation is as important as the final result.

Temporal Understanding: By treating sketching as a temporal process, this work highlights the importance of sequence modeling in creative domains and suggests that video models may be underutilized for tasks beyond traditional video generation.

Scalability Questions: While the minimal data requirement is impressive, it raises questions about how performance scales with additional training data and whether there are diminishing returns given the strong pretrained priors.

The work also points toward broader applications of video models in domains where temporal structure matters—from handwriting generation to dance choreography to procedural content creation.

Takeaways

1. Sketching should be modeled as a temporal process rather than a static image generation task, as the sequential nature of drawing carries important semantic and artistic information.

2. Pretrained video diffusion models contain powerful temporal priors that can be effectively adapted to specialized sequential generation tasks with minimal domain-specific training data.

3. Combining LLMs for semantic planning with video models for visual rendering creates a powerful hybrid architecture that leverages complementary strengths of both model types.

4. Decoupling stroke ordering learning from appearance learning through two-stage fine-tuning enables efficient use of both synthetic and limited human-generated training data.

5. Text-controlled stroke ordering enables new forms of controllability in sketch generation, with applications in education, animation, and interactive creative tools.

6. Extreme data efficiency (seven training examples) demonstrates that strong pretrained models can dramatically reduce the data requirements for specialized generative tasks.