VisionFoundry: Teaching Vision Models to See with Synthetic Data

Paper: 2604.09531 Authors: Guanyu Zhou, Yida Yin, Wenhao Chai, Shengbang Tong, Xingyu Fu, Zhuang Liu Categories: cs.CV, cs.AI, cs.CL

The Gap

Vision-language models like GPT-4V and Gemini can describe images fluently but stumble on basic visual perception: which object is closer? What’s the viewpoint angle? Prior work assumed this weakness came from model architecture or insufficient scale. This paper argues the real culprit is data starvation for low-level visual skills. Natural image datasets (COCO, ImageNet) contain rich semantic content but sparse supervision for spatial relationships, depth ordering, or viewpoint geometry. Existing synthetic data approaches require reference images, 3D models, or extensive human annotation, making them expensive and domain-specific.

Problem: VLMs fail at spatial reasoning
   |
   v
Hypothesis: Natural datasets lack low-level visual supervision
   |
   v
Method: Generate task-targeted synthetic data from keywords alone
   |
   v
Evidence: +7% MMVP, +10% CV-Bench-3D with 10K synthetic examples
   |
   v
Conclusion: Targeted synthetic supervision addresses the bottleneck

The Increment

One sentence: Before this paper, fixing VLM perception required expensive human annotation or domain-specific 3D assets; after, you can generate targeted training data from a task name alone.

Core Mechanism

VisionFoundry is a four-stage pipeline. Stage 1: An LLM takes a task keyword (e.g., “Depth Order”) and generates diverse question-answer pairs about that visual skill. Stage 2: The same LLM converts each QA pair into a text-to-image prompt that, when rendered, should produce an image where the answer is verifiable. Stage 3: A text-to-image model (FLUX) generates the image. Stage 4: A proprietary VLM (GPT-4o) verifies consistency—does the generated image actually support the intended answer? Only verified triples enter the dataset.

Task Keyword --> [LLM] --> QA Pairs
                            |
                            v
                    [LLM] --> T2I Prompts
                            |
                            v
                    [FLUX] --> Synthetic Images
                            |
                            v
                    [GPT-4o] --> Verified Triples
                                      |
                                      v
                              VisionFoundry-10K

Think of VisionFoundry as a custom textbook printing press. You tell the press what skill you want to teach (the task keyword). The press’s editor (LLM) writes practice problems and answer keys. The illustrator (T2I model) draws diagrams for each problem. The quality control inspector (verification VLM) checks that each diagram actually matches its problem—if the problem asks “which ball is closer?” but the diagram shows them equidistant, it’s rejected. Only verified pages go into the textbook. The student (target VLM) learns by working through these custom-made exercises.

Key Concepts

• Task-aware synthesis: Traditional synthetic data generation starts with 3D scenes or templates, then derives questions. VisionFoundry inverts this: start with the question type you want to teach, then generate images that answer it. This ensures every training example directly targets the skill gap. It’s like designing a workout routine for a specific muscle group rather than doing random exercises and hoping they help.

• Self-verification loop: The pipeline uses a strong VLM (GPT-4o) to filter out inconsistent examples where the image doesn’t support the answer. This is crucial because text-to-image models don’t guarantee spatial accuracy—FLUX might generate “red ball in front of blue ball” with reversed depth. The verification step catches these failures without human review. Empirically, about 60% of generated triples pass verification, which is high enough to be practical.

Framework Shift

Before (mainstream approach):        After (this paper):

Natural Images                       Task Keyword
     |                                    |
     v                                    v
Human Annotation                     LLM Generation
     |                                    |
     v                                    v
Generic VQA Dataset                  T2I Synthesis
     |                                    |
     v                                    v
Train VLM                            VLM Verification
     |                                    |
     v                                    v
Hope perception improves             Targeted Dataset
                                          |
                                          v
                                     Train VLM
                                          |
                                          v
                                  Measured perception gains

From passive data collection to active curriculum design, the core shift is treating data generation as a teaching problem, not a sampling problem.

Expert Assessment

Problem choice: This is a real gap. The observation that VLMs struggle with spatial reasoning despite strong semantic understanding has been documented (MMVP benchmark, CV-Bench), but the field defaulted to “scale up pretraining” rather than questioning data composition. The hypothesis that natural images provide insufficient low-level supervision is plausible and testable.

Method maturity: The pipeline is straightforward—chain LLMs and T2I models with a verification step. The cleverness is in the problem formulation, not the technique. However, there’s a dependency on proprietary models (GPT-4o for verification, FLUX for generation) that limits reproducibility. The paper doesn’t explore whether open-source alternatives (Llama, SDXL) would work, which feels like a missed opportunity.

Experimental integrity: Baselines are fair. They compare against natural data (VQAv2) and show that synthetic data alone outperforms it on perception tasks while preserving general capabilities (MMBench, SEED-Bench). The scaling experiments (1K to 10K examples) show consistent improvement, which strengthens the claim. One concern: the verification step uses GPT-4o, and the target models are also trained to align with GPT-4o’s judgments. This could introduce circularity—are we teaching perception or teaching “what GPT-4o thinks is correct perception”?

Writing quality: The paper is clear and well-structured. The weakest section is the related work—it lists prior synthetic data efforts but doesn’t deeply engage with why they failed to address this specific gap. The ablation studies (Section 4.3) are thorough, testing each pipeline component. The failure case analysis (Section 4.4) is honest about limitations, which is refreshing.

Verdict: weak accept — The core insight (targeted synthetic supervision for low-level vision) is valuable and the results are convincing, but the reliance on proprietary models and potential circularity in verification warrant caution.

Takeaways

For practitioners: If you’re training a VLM and notice it fails on a specific visual skill (e.g., counting, occlusion reasoning), don’t just add more general data. Use this pipeline structure: (1) generate targeted QA pairs for that skill, (2) synthesize images with T2I models, (3) verify consistency with a strong model. Even a few thousand targeted examples can outperform millions of generic ones.

For researchers: The verification step is the unsung hero. Without it, T2I model errors would poison the dataset. This suggests a broader principle: when using generative models for data augmentation, always close the loop with a verification mechanism. The paper shows 60% pass rate is practical; below 30% you’re wasting compute.

Transferable technique: The “task keyword → LLM → synthetic data” pattern works beyond vision. You could apply this to generate targeted training data for any skill where you can programmatically verify correctness: logical reasoning (verify with a theorem prover), code generation (verify with unit tests), or structured prediction (verify with constraint checkers).