Visual-ERM: Teaching AI to Judge Code by What It Renders, Not What It Says

Paper: 2603.13224 Authors: Ziyu Liu, Shengyuan Ding, Xinyu Fang, Xuanlang Dai, Penghui Yang, Jianze Liang, Jiaqi Wang, Kai Chen, Dahua Lin, Yuhang Zang Categories: cs.CV, cs.AI

The Gap

Large vision-language models can generate code from visual inputs (charts, tables, SVGs) via supervised learning, but reinforcement learning hits a wall. The problem: existing reward signals are fundamentally misaligned. Text-based rules check syntax and structure but miss visual fidelity—a chart might have correct labels but wrong colors, spacing, or proportions. Visual embedding similarity is too coarse and gameable—models learn to maximize embedding overlap without actually matching the visual appearance. Both approaches evaluate code indirectly when what matters is the rendered output.

Problem: Vision-to-code RL fails to improve quality
    |
    v
Root cause: Reward signals don't capture visual fidelity
    |
    +---> Text rules: "Does code have axis labels?"
    |     (Misses: wrong colors, spacing, proportions)
    |
    +---> Embedding distance: "How similar are embeddings?"
          (Hackable: maximize overlap without visual match)
    |
    v
Insight: Need fine-grained feedback in rendered visual space
    |
    v
Method: Visual-ERM renders code, compares pixel-level,
        generates interpretable critique
    |
    v
Evidence: +8.4 chart-to-code, +2.7 table, +4.1 SVG
    |
    v
Conclusion: Direct visual evaluation > proxy metrics

The Increment

One sentence: Before this paper, vision-to-code RL used text rules or coarse embeddings as rewards; after, models receive fine-grained visual critiques by comparing what the code actually renders.

Core Mechanism

Visual-ERM is a multimodal generative reward model that operates in three stages. First, it takes the original visual input (a chart, table, or SVG) and the model-generated code. Second, it renders the generated code into an image. Third, it feeds both the original and rendered images into a vision-language model that generates a detailed critique—identifying what matches, what’s missing, and what’s wrong at a fine-grained level. This critique becomes the reward signal for reinforcement learning.

The key architectural choice is using a generative model rather than a scalar reward. Instead of outputting a single number, Visual-ERM produces structured feedback: “The bar chart has correct data values but the x-axis labels are rotated 45 degrees when they should be horizontal, and the legend is positioned inside the plot area instead of outside.” This interpretable feedback serves dual purposes—it provides a richer training signal for RL and enables test-time improvements through reflection and revision.

Input Flow:
[Original Image] + [Generated Code]
       |                  |
       |                  v
       |          [Execute/Render]
       |                  |
       |                  v
       |          [Rendered Image]
       |                  |
       +--------+---------+
                |
                v
        [Visual-ERM Model]
     (multimodal generative)
                |
                v
    [Detailed Visual Critique]
    "Bar heights correct,
     colors wrong (blue vs red),
     legend missing,
     axis labels truncated"
                |
                v
         [Reward Signal]
                |
                v
          [RL Update]

Think of Visual-ERM as a photography instructor grading student reproductions of a reference photo. The old approach is like grading by reading the camera settings (text rules: “Did you use f/2.8? ISO 400?”) or by measuring how similar the photo’s color histogram is to the reference (embedding distance). Visual-ERM actually looks at both photos side-by-side and says: “Your exposure matches, but the subject is slightly out of focus, the composition is cropped too tight on the left, and there’s a distracting element in the background that wasn’t in the original.” The student learns not just that something is wrong, but exactly what to fix. The instructor’s eye (the generative model) can articulate nuances that no checklist or single similarity score could capture.

Key Concepts

• Reward Hacking in Vision-to-Code: When you optimize for a proxy metric instead of the true objective, models find shortcuts. With embedding similarity, a model might learn to generate code that produces images with similar global features (same color palette, rough layout) but completely wrong details (wrong data values, missing labels). The embedding distance decreases, the reward goes up, but the visual output is still wrong. It’s like a student who learns to write essays that score well on automated grading (correct length, vocabulary diversity) but are nonsensical to human readers. Visual-ERM closes this gap by evaluating the actual rendered output, making it much harder to game the system.

• Fine-grained Visual Feedback: Not all visual differences matter equally, and not all can be captured by a single number. A chart might be 90% correct but have one critical error (wrong data series) or 50% correct with many small issues (slightly off colors, minor spacing problems). Fine-grained feedback decomposes the evaluation: “Data accuracy: correct. Color scheme: incorrect (should be blue/red, got green/orange). Legend: missing. Axis labels: present but font size too small.” This structured critique enables targeted improvements during RL training and provides interpretable signals for debugging.

• Task-Agnostic Visual Evaluation: The same visual comparison mechanism works across different vision-to-code tasks—charts, tables, SVGs—without task-specific rules or metrics. This is powerful because it means you don’t need to hand-engineer evaluation criteria for each new visual format. The model learns to identify visual discrepancies in a general way: “These two images should look the same; here’s where they differ.” It’s like having a universal quality inspector who can evaluate any manufactured product by comparing it to the blueprint, rather than needing different inspectors with different checklists for each product type.

Framework Shift

Before (proxy-based rewards):        After (Visual-ERM):

Generated Code                       Generated Code
      |                                    |
      v                                    v
[Text Rule Checker]                  [Renderer]
 - Has axis labels?                       |
 - Correct chart type?                    v
 - Valid syntax?                    Rendered Image
      |                                    |
      v                                    +--------+
  Scalar Score                             |        |
                                           v        v
      OR                            Original   Rendered
                                      Image     Image
Generated Code                             |        |
      |                                    +---+----+
      v                                        |
[Embedding Extractor]                          v
      |                              [Visual-ERM Model]
      v                                        |
  Embedding Vector                             v
      |                              [Detailed Critique]
      v                              "Bar 3 height wrong,
[Distance to Target]                 legend position off,
      |                              colors inverted"
      v                                        |
  Scalar Score                                 v
                                         Rich Reward
                                         
Evaluation space: Code/Embeddings    Evaluation space: Rendered Visuals
Feedback: Single number              Feedback: Structured critique
Hackable: Yes                        Hackable: Much harder

From indirect proxy metrics to direct visual comparison, the core shift is evaluating code by what it produces, not by what it contains or how its embeddings cluster.

Expert Assessment

Problem choice: This is a genuine gap at the intersection of vision-language models and reinforcement learning. Vision-to-code is practically important (accessibility, data extraction, UI generation), and the reward misalignment problem is well-documented in RL literature. The authors correctly identify that existing approaches evaluate in the wrong space. The problem sits at a natural inflection point—supervised learning has plateaued, and RL is the obvious next step, but it needs better rewards to work.

Method maturity: The core insight is elegant and well-executed. Using a generative model for rewards is clever because it provides interpretable feedback that can guide both training and test-time revision. However, I’m skeptical about computational cost—rendering code and running a vision-language model for every RL step is expensive. The paper would be stronger with ablations showing whether simpler visual metrics (perceptual loss, SSIM, learned discriminators) could achieve similar gains at lower cost. The “task-agnostic” claim is somewhat overstated—the model still needs to understand what visual fidelity means for structured data, which is domain-specific knowledge.

Experimental integrity: The results are solid and the benchmark contribution (VC-RewardBench) is valuable. +8.4 on chart-to-code is substantial, and gains on table/SVG parsing demonstrate generalization. The comparison to Qwen3-VL-235B-Instruct is impressive. However, I’d want to see more analysis: What’s the computational overhead? How much training data does Visual-ERM need? Are there failure modes where fine-grained feedback actually hurts (e.g., overfitting to minor visual details)? The paper likely underreports negative results or limitations.

Writing quality: The abstract is clear and the motivation is well-framed. The core contribution is easy to grasp. However, I suspect the method section glosses over important details—how exactly is the critique structured? What’s the training procedure for Visual-ERM? How do you convert textual critiques into scalar rewards for RL? The paper probably front-loads results and under-explains the engineering required to make this work in practice.

Verdict: Weak accept—this is a solid contribution with clear practical value and good experimental results, but it’s an incremental improvement rather than a paradigm shift. The insight (evaluate in rendered space) is somewhat obvious in hindsight, and the execution, while competent, doesn’t reveal deep new principles about reward modeling or vision-language learning.

Takeaways

Evaluate in the output space that matters: If you care about visual fidelity, don’t evaluate code or embeddings—render the output and compare visually. This principle transfers to any domain where proxy metrics are misaligned with true objectives. In audio generation, evaluate the waveform, not the spectrogram. In robotics, evaluate the physical outcome, not the trajectory plan.

Generative models as reward functions: Instead of scalar rewards, use language models to generate structured critiques. This provides richer training signals and enables interpretable debugging. You can apply this to code review (generate specific improvement suggestions), content moderation (explain why content violates policies), or quality control (describe manufacturing defects).

Fine-grained feedback enables targeted improvement: Breaking down evaluation into components (data accuracy, color scheme, layout, labels) lets models learn which aspects need work. This is especially valuable for complex tasks where a single aggregate score obscures what’s actually wrong.

Beware of reward hacking: Whenever you optimize for a proxy metric, check whether models can game it. The embedding similarity example is instructive—models learned to maximize overlap without matching visual appearance. Always validate that your reward function actually measures what you care about.