Diagram-MMU: Can MLLMs Actually Write the TikZ Behind a Figure?

Paper: 2608.12262 Authors: Weihao Bo, Shan Zhang, Yanpeng Sun, Jie Liu, Yongke Yao, Jinhao Du, Wei He, Kai Zou, Zechao Li, Jingdong Wang Categories: cs.CV, cs.AI

The Gap

Two separate literatures have grown up around figures, and neither one covers the thing people now actually do with MLLMs.

The first is diagram understanding as QA: AI2D, ScienceQA, ChartQA, MMMU-style science subsets. These ask text questions about an image and score the text answer. They measure whether a model can *read a figure, and by 2025 frontier models were largely saturating the easy end of them.

The second is image-to-code generation: Design2Code (screenshot to HTML), Plot2Code and ChartMimic (chart image to matplotlib), Image2Struct. These do ask for executable output, but the visual domain is charts and web pages — data-driven plots with a small grammar, or DOM layouts. Scientific *diagrams — architecture figures, circuit schematics, geometry constructions, pipeline blocks, commutative-style graphs — are a different beast: no underlying data table, arbitrary coordinates, hand-tuned anchors, node/edge topology that carries the meaning.

The third thing missing from both is the workflow. Tools like OpenAI Prism made “drop in a figure, get TikZ back” a default expectation, and in real use nobody does it in one shot: you generate, look at the render, and then edit. Nobody had a benchmark for the edit step, and nobody had measured whether letting the model compile and look at its own output actually helps.

   PROBLEM
   [ diagram-to-TikZ is now a real product feature ]
   [ but benchmarks test QA or chart-to-plot only  ]
            |
            v
   ASSUMPTION
   [ "understands a diagram" and "can reconstruct  ]
   [  a diagram as code" are DIFFERENT abilities   ]
   [  and should be scored separately              ]
            |
            v
   METHOD
   [ Diagram-MMU: 3.7k curated scientific diagrams ]
   [ 6 domains, 18.3k human-validated questions    ]
   [ 3 task heads:  parse | edit | QA              ]
   [ x2 modes:      direct | agentic (tools+loop)  ]
            |
            v
   EVIDENCE
   [ 12 MLLMs measured ]
   [ QA scores >> parse/edit scores                ]
   [ agentic: parse/edit UP , QA DOWN              ]
   [ one model (Claude-4.6 Opus) up on all three   ]
            |
            v
   CONCLUSION
   [ reasoning-over-diagrams is not the bottleneck; ]
   [ symbol-level reconstruction is.                ]
   [ agentic scaffolding is task-selective, not     ]
   [ a free upgrade.                                ]

The Increment

One sentence: Before, “can this model handle scientific figures?” was answered with a QA score; after, it’s answered with three separate numbers — read it, rebuild it, revise it — and the middle two turn out to be where everything falls apart.

Core Mechanism

There is no trained model here. The mechanism is the benchmark’s construction and its scoring loop, so that’s what to look at.

The data side starts from real scientific diagrams paired with ground-truth TikZ source across six domains, curated rather than scraped wholesale, then annotated with questions that a human checked. That gives each item a triplet: a rendered image, the code that produced it, and a set of validated question-answer pairs. Having all three in one item is what lets a single corpus feed three different tasks instead of one.

The task side slices that triplet three ways. *Diagram-to-code parsing: give the image, ask for TikZ, compile it, compare the render to the original. Diagram-to-code editing: give the image plus its code plus an instruction (“make the arrows dashed”, “swap these two blocks”), ask for revised code, compile, check that the requested change happened and nothing else broke. Diagram QA: give the image, ask the validated questions, score the text. Parsing tests full reconstruction, editing tests localized symbol grounding — you must find which line of code corresponds to the thing being pointed at — and QA tests reading.

The mode side is the second axis. In the direct mode the model emits an answer in one pass. In the agentic mode it gets a loop: emit code, compile it, look at the render, diff against the target, revise. That loop is exactly the thing a human does, and it is precisely where the paper’s most interesting result lives — it helps the code tasks and *hurts QA, which tells you the loop is not “more thinking is better,” it’s a specific tool that fits specific tasks.

   ONE CORPUS ITEM
   +---------------------------------------------+
   |  image.png   <---- compiled from ---- code  |
   |      |                                 |    |
   |      +------ human-validated Q/A ------+    |
   +---------------------------------------------+
        |              |                |
        v              v                v
   [ PARSE ]       [ EDIT ]          [ QA ]
   img -> code    img+code+instr    img+q -> text
                   -> code'

   MODE A: direct
   input --> model --> answer --> score

   MODE B: agentic
   input --> model --> code --> [ compile ] --+
               ^                    |          |
               |                    v          |
               +---- diff <---- render <-------+
                     (n rounds)
                        |
                        v
                      score

   OBSERVED
   parse : direct < agentic
   edit  : direct < agentic
   QA    : direct > agentic     (loop backfires)

The metaphor: a music transcription exam. The diagram is a recording. The TikZ source is the sheet music. Three exam stations, one recording:

At the listening station (QA) they play the clip and ask “what key is it in? how many voices? does the melody rise or fall at the end?” Strong ears pass easily. This is where models already do well — they hear the structure.

At the transcription station (parsing) they play the clip and hand you blank staff paper. Now hearing isn’t enough; you need to commit to exact pitches, exact rhythms, exact bar lines. A musician with a great ear and shaky notation skills fails here, and that is exactly the profile the paper finds in MLLMs: they can describe the figure but cannot write down the score that produces it.

At the revision station (editing) they give you the recording *and the existing score, and say “make bar 12 minor.” You don’t need to transcribe from scratch — you need to locate bar 12 in the notation. That’s grounding, not generation, and it’s a genuinely different skill.

The agentic mode is letting candidates sit at a piano and play back what they wrote. For transcription and revision this is obviously useful — you hear your mistake and fix the note. But for the listening station, walking to the piano is a distraction: you spend your attention operating an instrument instead of listening, and you score worse than if you’d just answered. Only the best candidate in the room uses the piano well at all three stations.

Key Concepts

• Parsing vs. captioning (the perception-to-symbol inversion): Describing a picture is a many-to-one map — lots of correct sentences describe the same figure, and being roughly right is fine. Producing the source code is a one-to-many-but-checkable map — the output has to compile, coordinates have to be near-consistent, and a rendered comparison is unforgiving. It’s the difference between “this looks like a box with three arrows going into it” and stating the box’s position, size, and each arrow’s start and end anchor. The first is a summary; the second is an *inversion of the rendering process. Models were trained on oceans of the first and comparatively little of the second, which is a satisfying explanation for the gap the paper measures.

• Editing as a grounding probe: Suppose you get an instruction like “make the loss arrow red.” To do that you must map a visual object (that arrow) to a specific token span in the code (the \draw line that produced it). If you can’t localize, you’ll either rewrite the whole file — usually breaking something else — or edit the wrong line. This makes editing a cheap and surprisingly sharp diagnostic: the pass/fail signal isolates grounding, because the model was handed the code and didn’t have to invent it. Whether the metric properly punishes collateral damage (edit succeeded but the rest of the figure moved) is the part I’d most want to read the details on.

• Tool loops are not monotone improvements: The instinct is “let the model use tools and it gets better.” The paper’s asymmetric result says otherwise: the same scaffolding lifts code tasks and depresses QA. Mechanically this is unsurprising once you say it out loud — the loop injects long tool transcripts and rendered intermediates into a context where the answer was already available in the first glance, so you get context dilution plus opportunities to talk yourself out of a correct answer. The practical version: agent harnesses need per-task gating, not a global on-switch.

Framework Shift

 Before (mainstream approach):        After (this paper):

 [ diagram image ]                   [ diagram image ]
        |                              |     |     |
        v                              v     v     v
 [ MLLM ]                           parse  edit   QA
        |                              |     |     |
        v                              v     v     |
 [ text answer ]                    [ TikZ code ]  |
        |                              |            |
        v                              v            |
 ( accuracy % )                    [ compile ]      |
                                       |            |
 one number, one axis                  v            |
                                   [ render ]       |
 ( or, separately: )                   |            |
 [ chart image ]                       v            v
        v                          ( visual diff )( acc )
 [ matplotlib code ]
        v                          x  \{ direct , agentic \}
 ( plot similarity )               = 6 measurements per model

 charts, not diagrams              closed loop, three abilities

From “score the answer” to “score the reconstruction, the revision, and the answer separately — under both one-shot and closed-loop conditions,” the core shift is treating figure competence as a small matrix of distinct skills rather than a single scalar, with the compiler as the referee.

Expert Assessment

Problem choice: Real gap, well-timed, slightly opportunistic. The workflow being benchmarked — paste figure, get TikZ, iterate — genuinely became a mainstream use case, and scientific diagrams genuinely are underrepresented next to charts and screenshots. The parse/edit/QA decomposition is the right decomposition, and editing is the piece nobody had. The deduction is that leading with “OpenAI Prism” in the abstract is product-cycle framing that will read as dated within a year, and naming a specific frontier model in the abstract’s headline finding guarantees the paper’s most quoted sentence expires quickly. The finding underneath (“agentic scaffolding is task-selective”) is durable; the packaging is not.

Method maturity: This is careful assembly rather than clever insight, which is fine for a benchmark. The real difficulty in this area is metrics, and that’s where I’d apply pressure. Diagram-to-code scoring in prior work has leaned on compile rate, pixel/embedding similarity of renders, code-level overlap, and LLM-as-judge — each with a known failure mode. Pixel similarity punishes a legitimately cleaner layout; code overlap rewards imitating the ground-truth author’s style over producing an equivalent figure; LLM judges drift and favor verbose output. The abstract doesn’t tell us which combination is used, and the credibility of the headline “parsing is harder than QA” claim rests almost entirely on that choice, since it’s a comparison across two metric families rather than within one. A simpler alternative the field keeps under-using: score parsing by *re-asking the QA questions against the reconstructed render. That makes parse and QA commensurable on one scale and sidesteps style-matching entirely.

Experimental integrity: 12 models with a direct/agentic split is a reasonable sweep, and reporting a result that cuts against the “agents always help” narrative is a good sign — that’s a finding you keep only if you didn’t tune to get it. Two things I’d want checked. First, contamination: TikZ source for arXiv figures is on arXiv, so if the diagrams come from public papers, some ground-truth code may be in pretraining data, which inflates parsing scores unevenly across model vintages; a held-out or newly-authored subset would settle it. Second, the agentic comparison needs matched budgets — if the agentic runs get more tokens and more attempts, “agentic helps parsing” partly restates “more compute helps,” and the QA regression is then the more informative half of the result. The single-model-improves-everywhere claim I’d treat as an anecdote until it’s shown to be a capability rather than a harness fit.

Writing quality: The abstract spends its opening on a product and its closing on a leaderboard, and spends nothing on how anything is scored or how “human-validated” was operationalized. Rewriting the evaluation-protocol section to state, per task, exactly what is measured, what a failure mode looks like, and what the human-agreement rate on the validation was would raise the whole paper a tier — because for a benchmark, the protocol *is the contribution. A per-domain breakdown across the six domains would also earn its space: “circuits are hard, block diagrams are easy” is more actionable to a method-builder than an aggregate ranking.

Verdict: weak accept — the parse/edit/QA decomposition and the asymmetric agentic result are genuine contributions to a real gap, but the value hinges on metric design that the abstract leaves unspecified, and the framing is tied to a product moment that will age fast.

Takeaways

Things worth stealing:

• Edit-a-given-artifact as a grounding test. Handing the model the existing source and asking for a targeted change isolates “can you locate the thing” from “can you build the thing.” This transfers anywhere you have a render-from-source pipeline: SVG, Mermaid, CAD scripts, Manim, shader code, Verilog with a waveform, Terraform with a topology view. It’s cheaper to annotate than full generation ground truth and it fails more informatively.

• Score generation by round-tripping through comprehension. If you’re evaluating any image-to-code task, consider asking your QA questions against the *reconstructed render rather than diffing pixels. Semantic-equivalence-by-behavior avoids penalizing valid stylistic divergence, and it makes generation and understanding scores directly comparable.

• Gate your tool loops per task. The concrete lesson for anyone shipping an agent: verification loops pay off where output is executable and checkable, and cost you where the answer was already visible in one look. Don’t ship one global agent mode — route. And if you’re benchmarking your own agent, always report the non-agentic baseline at matched budget, or you’ll mistake compute for capability.

• A caution for interpreting your own eval wins. Diagram-to-code is a domain where ground truth is publicly available alongside the images. If your model looks unusually strong at reconstructing published figures, check for contamination before you believe it.