VFIG: Teaching Vision Models to Redraw Technical Figures as Editable Vectors

Paper: 2603.24575 Authors: Qijia He, Xunmei Liu, Hammaad Memon, Ziang Li, Zixian Ma, Jaemin Cho, Jason Ren, Daniel S Weld, Ranjay Krishna Categories: cs.CV, cs.AI

The Gap

Existing vectorization methods (Potrace, VTracer) work well for simple shapes but collapse on complex technical diagrams with nested structures, text annotations, and precise geometric relationships. Recent learning-based approaches (Im2Vec, DiffVG) either require ground-truth vector supervision that doesn’t exist at scale or produce outputs that lack semantic editability. The core problem: treating vectorization as pixel-level reconstruction misses the hierarchical compositional structure that makes SVG useful in the first place.

Problem: Rasterized figures → Lost editability
                |
                v
Prior approaches: Pixel-matching optimization
                |
                +---> Fails on complex diagrams
                +---> No semantic structure
                |
                v
This paper: Hierarchical structure learning
                |
                +---> Primitives first (SFT)
                +---> Global coherence (RL)
                |
                v
Evidence: 66K dataset + VFIG-BENCH metrics
                |
                v
Conclusion: Matches GPT-5.2 on complex figures

The Increment

One sentence: Before VFIG, you either manually redrew technical figures or got unusable vector outputs; after VFIG, you get editable SVG with preserved semantic structure from a single image.

Core Mechanism

VFIG operates in two stages. First, supervised fine-tuning teaches the model atomic SVG primitives (paths, circles, text) by training on isolated elements. The model learns to map visual features to SVG code tokens autoregressively, building a vocabulary of geometric operations. Second, reinforcement learning optimizes for global properties that supervised learning can’t capture: layout consistency across the entire diagram, topological correctness of connections, and visual fidelity measured by perceptual metrics.

The data pipeline is equally important. VFIG-DATA combines 66K figure-SVG pairs from two sources: real academic paper figures (scraped and filtered for quality) and procedurally generated diagrams (flowcharts, network graphs, technical schematics). The procedural generation ensures coverage of edge cases and compositional patterns that real data alone wouldn’t provide.

Input Image (PNG/JPEG)
        |
        v
   Vision Encoder
        |
        +---> Visual features
        |
        v
   Language Decoder (autoregressive)
        |
        +---> SVG tokens: <path d="M10,20 L30,40"/>
        |
        v
   Stage 1: SFT on primitives
        |     (circle, line, text → correct SVG)
        |
        v
   Stage 2: RL refinement
        |     Reward = layout + topology + fidelity
        |
        v
   Output: Editable SVG code

Think of VFIG like teaching someone to transcribe sheet music. First, you drill individual notes and symbols (supervised learning of primitives). The student learns “this squiggle means treble clef, that dot means staccato.” But knowing symbols doesn’t mean you can transcribe a Beethoven sonata—you need to understand phrasing, dynamics, how measures relate. That’s the RL phase: optimizing for musical coherence across the entire piece. The student gets feedback not on individual notes but on whether the transcription captures the composition’s structure. VFIG does the same: primitives first, then global diagram integrity.

Key Concepts

• Coarse-to-fine curriculum: Start with easy examples (simple shapes, few elements) and gradually increase complexity (nested groups, overlapping objects, dense layouts). This prevents the model from getting stuck in local minima where it learns to copy pixels instead of understanding structure. It’s like learning to draw: you sketch rough shapes before adding details, not the other way around.

• Reinforcement learning for structure: Supervised learning optimizes token-by-token prediction, but SVG quality depends on properties that span the entire output—does the flowchart’s arrow actually connect the right boxes? Are parallel lines truly parallel? RL uses a reward function that measures these global properties, letting the model explore variations and learn what “good diagram structure” means beyond pixel matching.

• Procedural data augmentation: Real-world SVG-image pairs are scarce and biased toward certain diagram types. Procedural generation creates synthetic training data by programmatically building diagrams (random graph layouts, parameterized flowcharts) and rendering them. This gives the model exposure to compositional patterns and edge cases that real data doesn’t cover, like deeply nested groups or unusual geometric configurations.

Framework Shift

Before (pixel reconstruction):        After (structural decomposition):

Image → Optimize paths                Image → Parse hierarchy
  |       to match pixels               |       primitives + relations
  |                                     |
  +---> Local minima                    +---> Atomic elements (SFT)
  +---> No editability                  +---> Global coherence (RL)
  |                                     |
  v                                     v
Unusable vector soup                  Editable semantic SVG

From pixel-level optimization to hierarchical structure learning, the core shift is treating vectorization as compositional understanding rather than visual mimicry.

Expert Assessment

Problem choice: Real and underappreciated. The “lost source file” problem is endemic in academic publishing and technical documentation. Prior work focused on simple logos or handwriting, but complex diagrams are where the pain actually lives. This sits at the intersection of document understanding and generative modeling—a natural next step after OCR and layout analysis matured.

Method maturity: The two-stage curriculum is well-motivated, but the RL reward function feels hand-engineered. They use a weighted combination of perceptual loss, topology metrics, and layout consistency—each requiring manual tuning. A learned reward model (like RLHF) might generalize better. The procedural generation is clever but risks distribution mismatch: synthetic flowcharts don’t capture the idiosyncrasies of real hand-drawn diagrams.

Experimental integrity: Baselines are fair (Potrace, Im2Vec, GPT-4V), but the comparison with GPT-5.2 is murky—they don’t specify prompting strategy or whether GPT-5.2 was fine-tuned. VFIG-BENCH metrics are novel but not validated against human judgment at scale. The VLM-Judge score (0.829) is a proxy, not ground truth. I’d want to see inter-annotator agreement and failure case analysis.

Writing quality: The method section is dense—Figure 3 tries to show the entire pipeline but ends up cluttered. The ablation study (Table 3) is buried in the appendix when it should be front and center. The related work undersells how much this builds on recent VLM architectures (Flamingo, BLIP-2). Rewriting Section 4.2 to separate data curation from model training would clarify the contributions.

Verdict: weak accept — Solid execution on a real problem, but the method feels like careful engineering rather than a conceptual breakthrough. The dataset is the most reusable contribution.

Takeaways

The coarse-to-fine curriculum is immediately applicable to any structured generation task (code, markup, formal languages). Start with atomic elements, then optimize for global coherence—don’t try to learn both simultaneously. The procedural data generation strategy is also transferable: when real paired data is scarce, programmatically generate synthetic examples that cover compositional edge cases. Finally, the insight that RL can optimize structural properties that supervised learning misses applies beyond vectorization—think protein folding, circuit design, or any domain where local correctness doesn’t guarantee global validity.