LottieGPT: Teaching AI to Generate Editable Vector Animations

Paper: 2604.11792 Authors: Junhao Chen, Kejun Gao, Yuehan Cui, Mingze Sun, Mingjin Chen, Shaohui Wang, Xiaoxiao Long, Fei Ma, Qi Tian, Ruqi Huang Categories: cs.CV

The Gap

Video generation models (Sora, Runway, Pika) have made impressive strides, but they all output raster pixels. Meanwhile, a huge chunk of the internet’s animations—loading spinners, UI transitions, explainer videos, stickers—are vector-based (SVG, Lottie). These formats are resolution-independent, tiny in file size, semantically structured, and editable by designers. Current generative models can’t produce them at all. They’re stuck in pixel land.

The prior work on vector graphics generation (SVG-VAE, DeepSVG, IconShop) handles only static images or simple shapes. They don’t model temporal dynamics, layered composition, or keyframe-based motion. Large multimodal models have shown they can generate structured outputs (3D meshes, code, layouts), but nobody has tackled vector animation.

Problem: Video models output pixels, not editable vectors
   |
   v
Assumption: Vector animation can be treated as structured sequence
   |
   v
Method: Tokenize Lottie format + finetune multimodal LLM
   |
   v
Evidence: 660K dataset + autoregressive generation works
   |
   v
Conclusion: First native vector animation generation from prompts

The Increment

One sentence: Before this paper, generative models could only output raster video; after it, they can generate resolution-independent, editable vector animations directly from text or images.

Core Mechanism

LottieGPT has three components. First, a Lottie Tokenizer that converts JSON animation files into discrete tokens. Lottie files describe animations as layers of shapes (rectangles, ellipses, paths) with transforms (position, rotation, scale) and keyframe-based motion curves. The tokenizer flattens this hierarchical structure into a linear sequence, using special tokens to mark layer boundaries, shape types, and temporal keyframes. It quantizes continuous values (coordinates, colors, timing) into discrete bins to make them tokenizable.

Second, a massive dataset: LottieAnimation-660K, scraped from design platforms and animation libraries. It contains 660K animated Lottie files and 15M static ones. This is orders of magnitude larger than prior vector graphics datasets, which had at most a few thousand samples.

Third, the model itself: they finetune Qwen-VL, a large vision-language model, on sequences of (prompt, tokenized Lottie) pairs. The model learns to autoregressively predict the next token in the Lottie sequence, conditioned on text descriptions or reference images. At inference, you give it a prompt like “a bouncing ball” or an image of a logo, and it generates a complete Lottie animation token by token, which you then decode back into a playable JSON file.

Input (text/image) 
   |
   v
[Qwen-VL Encoder] --> Context embedding
   |
   v
[Autoregressive Decoder] --> Token_1, Token_2, ..., Token_N
   |
   v
[Lottie Detokenizer] --> JSON animation file
   |
   v
Output: Editable vector animation

Think of it like teaching a language model to write music notation instead of audio waveforms. Music notation (vector animation) is symbolic, editable, and resolution-independent—you can transpose it, change instruments, or stretch tempo without quality loss. Audio waveforms (raster video) are just raw samples—you can’t easily edit the melody or change the key. The tokenizer is like converting sheet music into a sequence of words (“quarter note C4, half note E4, rest…”). The dataset is a massive library of sheet music in a standardized format. The model learns the grammar of music notation by predicting the next symbol. When you ask it to “compose a cheerful melody,” it writes out notation, not audio samples. You can then render that notation with any instrument or at any tempo.

Key Concepts

• Lottie Format: A JSON-based animation standard used widely on the web (Airbnb created it). Instead of storing pixel frames, it stores geometric primitives (shapes), their properties (color, stroke width), and keyframe animations (position at time 0, position at time 1). Think of it as SVG with a timeline. A loading spinner might be defined as “a circle that rotates 360 degrees over 1 second, looping forever.” The file is tiny (a few KB) and renders crisply at any resolution. Designers export animations from After Effects to Lottie, and developers embed them in apps or websites.

• Autoregressive Tokenization: Breaking structured data into a sequence of discrete tokens that a language model can predict one-by-one. For Lottie, this means converting nested JSON objects into a flat token stream. A rectangle might become: [LAYER_START] [SHAPE_RECT] [WIDTH_100] [HEIGHT_50] [COLOR_RED] [KEYFRAME_0] [POS_X_10] [POS_Y_20] [KEYFRAME_1] [POS_X_100] [POS_Y_20] [LAYER_END]. The model learns that after [SHAPE_RECT], you usually see dimension tokens, then color, then keyframes. It’s like teaching a model to write HTML by predicting tags and attributes in order.

• Semantic Alignment: Ensuring that token sequences preserve the logical structure of the original data. A naive tokenizer might just split the JSON string character-by-character, but then the model has to learn JSON syntax from scratch. LottieGPT’s tokenizer respects the semantic boundaries—it groups related properties together, uses special tokens for hierarchy, and quantizes values in a way that similar animations have similar token sequences. This makes the learning problem tractable. It’s the difference between teaching someone to write code by showing them raw bytes versus showing them functions, variables, and control flow.

Framework Shift

Before (mainstream approach):        After (this paper):

Text/Image Prompt                    Text/Image Prompt
      |                                    |
      v                                    v
[Diffusion Model]                    [Multimodal LLM]
      |                                    |
      v                                    v
Pixel Grid (H x W x T)               Token Sequence
      |                                    |
      v                                    v
Raster Video File                    [Detokenizer]
(MP4, WebM)                                |
                                           v
- Fixed resolution                   Lottie JSON File
- Large file size                          |
- Not editable                             v
- Lossy compression                  Vector Animation
                                           |
                                     - Resolution-independent
                                     - Tiny file size
                                     - Fully editable
                                     - Lossless

From pixel prediction to structured sequence generation, the core shift is treating animation as a symbolic language rather than a visual signal.

Expert Assessment

Problem choice: This is a real gap with practical impact. Vector animations dominate UI/UX design, motion graphics, and web content, yet generative models ignore them entirely. The problem sits at the intersection of computer graphics and generative AI—a natural next step after static SVG generation, but nobody took it seriously because Lottie files are complex and datasets didn’t exist. The authors manufactured the dataset themselves, which is both a strength (they solved the data problem) and a weakness (it raises questions about data quality and diversity).

Method maturity: The approach is straightforward: tokenize, scale up data, finetune a big model. There’s no novel architecture or training trick—it’s mostly engineering and data curation. That’s fine; sometimes the insight is “this problem is solvable with existing tools if you set it up right.” The tokenizer design is thoughtful (semantic alignment, quantization schemes), but it’s not groundbreaking. The real contribution is showing that autoregressive generation works for this modality at all.

Experimental integrity: Baselines are weak. They compare against SVG generation models (which don’t handle animation) and raster video models (which they convert to vector, introducing artifacts). There’s no prior work on vector animation generation, so they’re mostly comparing against themselves. The human evaluation is small-scale (50 samples). Quantitative metrics (FID, token efficiency) are reasonable but don’t capture editability or semantic coherence. The paper would be stronger with ablations on tokenizer design choices and failure case analysis.

Writing quality: The paper is well-structured and clear, but the evaluation section feels rushed. They spend pages on dataset construction (good) but gloss over failure modes and limitations. The qualitative results look impressive, but we don’t see how often the model produces broken JSON or semantically nonsensical animations. The related work section is thorough, but the discussion of why Lottie over other vector formats (SVG, Flash) is superficial.

Verdict: Weak accept — it’s the first work on an underexplored problem with a solid dataset contribution, but the method is incremental and the evaluation could be more rigorous.

Takeaways

The tokenizer design is the transferable idea here. If you’re working with any structured, hierarchical data (CAD files, music notation, game levels, UI layouts), the principles apply: (1) respect semantic boundaries when tokenizing, (2) use special tokens to mark hierarchy and object types, (3) quantize continuous values in a way that preserves similarity, (4) flatten the structure but keep it recoverable. The paper also demonstrates that you don’t need a custom architecture—finetuning a large multimodal model works if you frame the problem as sequence prediction. Finally, the dataset construction pipeline (scraping, filtering, deduplication, quality checks) is reusable for anyone building a new modality dataset from scratch.