QCalEval: Teaching Vision Models to Read Quantum Calibration Plots

Paper: 2604.25884 Authors: Shuxiang Cao, Zijian Zhang, Abhishek Agarwal, Grace Bratrud, Niyaz R. Beysengulov, Daniel C. Cole, Alejandro Gómez Frieiro, Elena O. Glen, Hao Hsu, Gang Huang Categories: quant-ph, cs.CV

The Gap

Quantum computing calibration requires interpreting experimental plots—resonance curves, Rabi oscillations, randomized benchmarking decay. Human experts read these plots to diagnose qubit behavior and tune parameters. Vision-language models (VLMs) are being deployed across scientific domains, but nobody has systematically tested whether they can actually understand quantum calibration data. Existing VLM benchmarks focus on natural images, charts from business contexts, or synthetic diagrams. The quantum calibration domain has unique characteristics: noisy experimental data, domain-specific visual patterns (Chevron patterns, oscillatory decay), and the need to reason across multiple related plots to diagnose issues.

Problem: VLMs used in quantum labs, but no evaluation exists
   |
   v
Assumption: Calibration plot understanding is testable via Q&A
   |
   v
Method: QCalEval benchmark (243 samples, 6 question types)
   |
   v
Evidence: Best model 72.3% zero-shot, ICL gap in open models
   |
   v
Conclusion: VLMs can handle single plots, struggle with multi-image reasoning

The Increment

One sentence: Before this paper, we had no idea if VLMs could interpret quantum calibration plots; after, we know they reach 72% accuracy on single-plot tasks but most open models degrade when reasoning across multiple images.

Core Mechanism

QCalEval is a benchmark dataset, not a new model architecture. The authors collected 243 calibration plot samples spanning 22 experiment families (Rabi oscillations, T1 decay, randomized benchmarking, etc.) from both superconducting qubits and neutral atom platforms. Each sample includes one or more plots and is paired with questions across six categories: reading values from plots, identifying patterns, comparing multiple plots, diagnosing issues, suggesting next steps, and explaining physical meaning.

The evaluation protocol tests two settings. Zero-shot: the model sees a plot and question with no prior examples. In-context learning (ICL): the model first sees 1-3 example plot-question-answer triplets, then answers a new question. The benchmark measures whether models can extract quantitative information (read a frequency from a peak), recognize qualitative patterns (identify a Chevron shape), and perform multi-step reasoning (compare two plots to diagnose which qubit has higher coherence).

Think of QCalEval as a standardized eye exam for VLMs in the quantum domain. Just as an eye chart tests whether you can read letters at different sizes and distances, QCalEval tests whether models can “read” different types of information from calibration plots at different levels of complexity. The single-plot questions are like reading large letters on the top row—most models pass. The multi-image reasoning questions are like reading the bottom row while also comparing what you see in your left eye versus right eye—this is where most models fail. The benchmark doesn’t teach models to see better; it reveals which models already have the visual reasoning capabilities needed for quantum calibration work.

Key Concepts

• Calibration plots: In quantum computing, you can’t directly observe a qubit’s state—you run experiments and measure outcomes. A calibration plot visualizes these measurements: the x-axis might be pulse duration, the y-axis might be qubit population in the excited state, and the curve shape tells you about the qubit’s properties. For example, a Rabi oscillation plot shows sinusoidal oscillations as you vary pulse length—the frequency of oscillation tells you the qubit’s coupling strength. Reading these plots requires recognizing domain-specific patterns (a Chevron pattern indicates two-qubit gate calibration) and extracting quantitative values (the peak frequency, the decay time constant). It’s not just “chart reading”—it’s interpreting noisy experimental data where the visual pattern encodes physical parameters.

• In-context learning (ICL) for vision: Standard ICL in language models means showing examples in the prompt. For vision-language models, ICL means showing example images with their answers before asking about a new image. The model must learn the task pattern from visual examples, not just text. QCalEval tests whether models can learn “if you see this curve shape, extract this value” from a few examples. The surprising finding: many open-weight models get worse with more examples, suggesting they struggle to transfer visual reasoning patterns across images. This is different from language ICL, where more examples almost always help.

• Zero-shot vs supervised fine-tuning gap: Zero-shot means the model has never seen quantum calibration plots during training—it must rely on general visual reasoning. Supervised fine-tuning (SFT) means explicitly training on calibration plot examples. The paper shows SFT improves zero-shot performance (the model learns to recognize quantum-specific patterns), but doesn’t close the ICL gap (the model still can’t learn new patterns from a few examples at inference time). This suggests two separate capabilities: recognizing learned patterns (improved by SFT) versus learning new patterns on the fly (a deeper architectural or training limitation).

Framework Shift

Before (implicit assumption):          After (QCalEval):

VLMs work on scientific plots          Systematic evaluation reveals:
         |                                      |
         v                                      v
   Deploy in labs                    Single-plot: 72% accurate
         |                           Multi-image: most models degrade
         v                                      |
   Hope for the best                            v
                                      Quantify the ICL gap
                                      Open models: -5 to -15 points
                                      Closed models: +10 to +20 points

From “VLMs probably work on quantum plots” to “VLMs handle single plots adequately but most cannot reason across multiple images,” the core shift is from assumption to measurement.

Expert Assessment

Problem choice: Real gap. Quantum computing is scaling rapidly, calibration is a bottleneck, and VLMs are being deployed without domain-specific evaluation. The paper addresses a practical need and reveals a genuine capability gap (multi-image reasoning) that wasn’t obvious from existing benchmarks. The timing is right—VLMs are mature enough for deployment but not yet systematically tested in this domain.

Method maturity: The benchmark design is solid but conservative. 243 samples is small by ML standards, though reasonable given the manual curation required. The six question types cover the right cognitive tasks. The ICL evaluation is well-designed—testing 1-shot, 2-shot, and 3-shot separately reveals the degradation pattern. However, the paper doesn’t deeply investigate why open models degrade under ICL. Is it a context window issue? Instruction following? Visual attention? The SFT ablation is useful but limited to one model scale (9B parameters). A more thorough ablation across scales and architectures would strengthen the claims.

Experimental integrity: Baselines are fair—testing both open and closed models, both general-purpose and domain-adapted. The numbers are credible: 72.3% for the best zero-shot model is neither suspiciously high nor artificially deflated. One concern: the paper doesn’t report inter-annotator agreement for the ground truth labels. Quantum calibration can be ambiguous (is that peak at 5.2 GHz or 5.3 GHz?), and without agreement metrics, we can’t assess label quality. The NVIDIA Ising Calibration 1 model is a nice contribution, though releasing only the final model (not training data or intermediate checkpoints) limits reproducibility.

Writing quality: The paper is clear and well-structured. The introduction motivates the problem effectively. The benchmark description is thorough. However, Section 4 (results) is dense with tables and lacks narrative flow—the reader must work hard to extract insights. The discussion of why open models degrade under ICL is superficial (one paragraph). Rewriting Section 4 to lead with key findings, then support with tables, would improve readability. The related work section is adequate but misses some relevant work on scientific figure understanding and multi-image reasoning in VLMs.

Verdict: weak accept — Addresses a real gap with a well-designed benchmark, but the analysis of why models fail is shallow and the dataset size is modest. Valuable as a first systematic evaluation, but leaves significant questions unanswered.

Takeaways

For practitioners deploying VLMs in scientific domains: Don’t assume zero-shot performance on single images transfers to multi-image reasoning. Test your specific use case. If your workflow requires comparing multiple plots or diagnosing issues across experiments, current open-weight models may not suffice—consider closed models or domain-specific fine-tuning.

For researchers building VLMs: The ICL degradation in open models is a red flag. It suggests a fundamental limitation in how these models handle visual context or instruction following with multiple images. This is a concrete research direction: why do more visual examples hurt performance, and how can we fix it?

For benchmark designers: QCalEval demonstrates that domain-specific benchmarks reveal capability gaps invisible in general benchmarks. The multi-image reasoning gap wasn’t apparent from existing VLM evaluations. When designing benchmarks, include tasks that require reasoning across multiple inputs, not just single-input understanding.

Transferable technique: The “example-based evaluation” approach (zero-shot vs 1-shot vs 2-shot vs 3-shot ICL) is a clean way to measure whether models can learn from visual examples. This protocol transfers to any domain where you want to test visual pattern learning—medical imaging, materials science, remote sensing.