SciDiagramEdit: Teaching AI to Edit Scientific Figures by Watching How Authors Revise

Paper: 2607.15272 Authors: Yasheng Sun, Zezi Zeng, Yifan Yang, Chong Luo, Wenyi Wang, Ziwei Liu, Jürgen Schmidhuber Categories: cs.CL, cs.AI

The Gap

We’ve gotten pretty good at editing photos with AI—things like “make the sky bluer.” But scientific figures are a different beast. They aren’t just pictures; they’re dense, structured arguments made of schematics, plots, text labels, and arrows, all following a strict visual grammar. Existing image editing models treat a figure like a monolithic photo, which doesn’t work. The fundamental limitation is that they operate on a flat, opaque raster image (like a PNG), losing the structural information of individual components. This paper tackles that exact boundary: how to edit a scientific diagram at the component level using a natural-language instruction.

Problem: Scientific figures are structured infographics, not photos.
         |
         v
Assumption: Editing must happen at the vector primitive level (labels, arrows, plots), not the pixel level.
         |
         v
Method: Mine real author revision pairs (before/after) from arXiv to train an agent that operates on editable vector sources.
         |
         v
Evidence: An agentic skill-evolution loop progressively improves edit accuracy on held-out revisions.
         |
         v
Conclusion: Natural paper revisions are a viable, high-quality training signal for complex, instruction-driven figure editing.

The Increment

One sentence: Before this paper, AI couldn’t reliably edit the structured components of a scientific diagram with a text command; after, it can learn to do so by observing how real authors revise their own figures.

Core Mechanism

The system has three main parts. First, it creates a new benchmark by mining the version histories of arXiv papers. It pairs figures from an author’s first draft with the revised version in a later draft, automatically extracting the “revision intent” as a natural language instruction (e.g., “Change the caption of panel (b) to ‘Loss comparison’”). This gives the AI a huge dataset of real-world, expert-performed edits.

Second, it doesn’t work on a final image file. It works on the editable source of the figure, which is often a LaTeX/TikZ or PowerPoint file. This source is parsed into a structured representation of its primitives (text boxes, lines, rectangles, charts) and their properties (position, color, text content). This turns a visual editing problem into a structured data manipulation problem.

Third, and most cleverly, it uses an “agentic learning via skill evolution” loop. Instead of just training a model once on the mined data, it has an agent that proposes edits, executes them on the source, checks the result against the target revision, and then refines its own editing skill specifications based on what it learns from success and failure. This iterative self-improvement is what lifts performance over multiple epochs.

[ arXiv Revision History ]
            |
            v
[ Mine Pairs (Before Fig, After Fig, Instruction) ] -> [ Benchmark ]
            |
            v
[ Parse Editable Vector Source (LaTeX/TikZ) ] -> [ Structured Primitive Graph ]
            |
            v
[ Agentic Editor ]
      ^       |
      |       v
[ Skill Evolution Loop ]
   (Propose -> Execute -> Compare -> Refine Skill Spec)
            |
            v
[ Improved Editing Agent ]

Think of it like a chef apprentice learning a new cuisine not from a textbook, but by watching a master chef repeatedly revise their own signature dish. Each revision (e.g., “add more chili, plate it differently”) is a concrete lesson. The apprentice doesn’t just memorize the final recipe; they learn the skill of making targeted adjustments based on feedback. Here, the “master chef” is the collection of all arXiv authors, their “revisions” are the mined before/after figure pairs, and the “skill” is the evolving set of editing operations the AI agent learns to apply.

Key Concepts

• Editable Vector Source: This is the recipe, not the baked cake. When you edit a PNG, you’re smearing icing on the finished product. When you edit the vector source (like the LaTeX code for a TikZ diagram), you’re changing the recipe itself—adjusting ingredient amounts, rearranging steps. SciDiagramEdit works at this recipe level, allowing precise, reversible changes to individual elements like a single label or arrow.
• Agentic Skill Evolution: This is the apprentice’s learning journal. A simple model would just memorize a mapping from “instruction + current state” to “action.” This paper’s model keeps a living document of its *skills—a refined specification of how to handle different types of edits. After each attempt, it reviews what worked and what didn’t, and rewrites its own skill guide. This makes the learning process more adaptive and generalizable.

Framework Shift

Before (mainstream approach):        After (this paper):
[ Photo Editor Mindset ]             [ Architect Mindset ]
      |                                    |
      v                                    v
[ Edit Raster Image (PNG) ]         [ Edit Blueprint (Vector Source) ]
      |                                    |
      v                                    v
[ Pixel-Level Manipulation ]        [ Primitive-Level Manipulation ]
      |                                    |
      v                                    v
[ Opaque, Brittle ]                 [ Structured, Co-editable ]

From treating figures as finished photographs to treating them as editable architectural blueprints, the core shift is moving the editing surface from pixels to primitives.

Expert Assessment

Problem choice: Excellent. Automating the tedious work of figure revision is a real, universally experienced pain point in research. It’s a niche that sits neatly at the intersection of NLP, computer vision, and scientific tooling, and has been largely ignored. This is a genuine gap.

Method maturity: Clever insight with a complex mechanism. The use of revision history as a self-supervised signal is the key insight and is quite elegant. The “skill evolution” loop adds complexity but seems necessary for the task’s diversity. A simpler approach—just fine-tuning a large multimodal model on the mined pairs—might have been a missed opportunity as a baseline.

Experimental integrity: The benchmark creation is innovative but introduces noise. Automatically extracting “revision intent” from a PDF diff is error-prone. The authors acknowledge this, but it’s a fundamental limitation. The held-out validation set is solid, but the final test relies on the same noisy signal. The baselines are fair, comparing against instruction-tuned LMMs and a direct editing model, showing clear improvements.

Writing quality: Clear and well-structured. The motivation is excellent. Where they could have elevated the paper is in the “Skill Evolution” section (Sec 3.3). The mechanism is the novel core, but the description feels a bit abstract. A more intuitive walkthrough or diagram of how the “skill specification” evolves would help immensely.

Verdict: weak accept — The problem is important and the approach of mining revisions is a valuable contribution, but the complexity of the agentic loop and reliance on noisy automatic supervision keep it from being a slam dunk. It opens a clear and promising research direction.

Takeaways

1. Mine the “Edit History”, not just the final product. For tasks involving iterative refinement (design, writing, coding), look for version control histories (Git, arXiv versions, Figma history). They are a rich, self-supervised signal for learning *how to edit, not just what the result looks like.
2. Push the edit interface down to the source. Whenever possible, design your AI system to operate on the structured, editable representation (code, vector graphics, document model) rather than a rendered output. This preserves structure and enables more precise, interpretable, and reversible edits.
3. Use iterative self-improvement for diverse tasks. For tasks where the space of possible actions is large and varied (like editing), a simple one-shot fine-tuning can plateau. An “evolutionary” loop where the model refines its own policy or specification based on execution traces can lead to significant gains in robustness and accuracy.