SciMDR: Teaching Models to Reason Across Full Scientific Papers

Paper: 2603.12249 Authors: Ziyu Chen, Yilun Zhao, Chengye Wang, Rilyn Han, Manasi Patwardhan, Arman Cohan Categories: cs.CL, cs.AI, cs.CV

The Gap

Scientific document understanding is genuinely hard to benchmark. You need three things at once: scale (enough data to train on), faithfulness (QA pairs that are actually grounded in the document), and realism (tasks that look like what a researcher actually does — reasoning across a full paper, not a cherry-picked paragraph). Prior work keeps dropping one of these balls.

Datasets like ScienceQA and MMMU are curated by hand — faithful and realistic, but tiny. Automated pipelines like those used in DocVQA or chart-QA generate at scale, but they either hallucinate answers or they test isolated snippets, not full-document reasoning. The model learns to answer “what does this figure show?” but not “given the methodology in section 3 and the results in figure 5, does the conclusion in section 6 hold?” That’s the gap: no large-scale training set that is simultaneously grounded, cross-modal, and document-scale.

Problem: Scale vs Faithfulness vs Realism -- can't have all three
    |
    v
Assumption: Synthesis at segment level is faithful;
            regrounding into full doc adds realism
    |
    v
Method: Stage 1 (Claim-Centric QA on segments)
        +
        Stage 2 (Document-Scale Regrounding)
    |
    v
Evidence: 300K QA pairs, 20K papers, SciMDR-Eval benchmark,
          fine-tuned models beat baselines on multiple benchmarks
    |
    v
Conclusion: The trade-off is breakable with a two-stage pipeline

The Increment

One sentence: Before this paper, you had to choose between a dataset that was large OR grounded OR realistic — SciMDR is the first to credibly claim all three at once.

Core Mechanism

The pipeline has two stages that solve different parts of the problem. Stage 1, Claim-Centric QA Synthesis, takes a focused segment of a paper — a figure with its caption, a table, a methods paragraph — and generates a QA pair plus an explicit reasoning chain that is tightly grounded in just that segment. Because the context window is small and controlled, hallucination is suppressed and faithfulness is high. The output is a clean, verifiable (question, answer, reasoning) triple.

Stage 2, Document-Scale Regrounding, takes those clean triples and programmatically re-embeds them back into the full paper. The model now has to find the relevant segment inside a 20-page document, integrate it with surrounding context, and produce the answer. The task difficulty jumps from “read this caption” to “navigate this paper.” The key insight is that the answer is still faithful — it was verified at segment level — but the retrieval and integration challenge is now realistic.

SciMDR-Eval is a separate expert-annotated benchmark built on top of this, used for evaluation only, not training. It tests whether the gains from fine-tuning on SciMDR actually transfer to held-out scientific reasoning tasks.

Input: Raw scientific papers (PDFs -> structured)
    |
    v
+----------------------------+
| Stage 1: Segment-level     |
|  - Extract focused segment |
|  - Generate (Q, A, Chain)  |
|  - Verify faithfulness     |
+----------------------------+
    |
    | clean (Q, A, Chain) triples
    v
+----------------------------+
| Stage 2: Document-level    |
|  - Re-embed into full doc  |
|  - Add distractor context  |
|  - Require cross-modal nav |
+----------------------------+
    |
    v
SciMDR: 300K QA pairs, 20K papers
    |
    v
Fine-tune foundation model
    |
    v
Evaluate on SciMDR-Eval + external benchmarks

Think of it like writing a good exam question. If you write the question while staring at the textbook page, you get a faithful question — but it’s too easy because students can just scan that one page. What this paper does is: write the question while looking at the page (Stage 1, faithful), then put the question on an exam where students have the whole textbook (Stage 2, realistic). The answer key is still correct because you wrote it from the source, but now the student has to actually know where to look. The two-stage structure decouples “is the answer right?” from “is the task hard enough?” — and that’s the load-bearing insight.

Key Concepts

• Claim-Centric QA: Instead of asking “what is in this document?” the synthesis targets a specific claim — a number in a table, a trend in a figure, a stated assumption in the methods. This keeps the generated QA pair anchored to something verifiable. Think of it as writing a citation: you’re not summarizing the paper, you’re pointing at one specific thing it says.

• Document-Scale Regrounding: After generating a QA pair from a segment, the pipeline re-inserts that pair into the full-document context. The question stays the same, but the model now has to locate the relevant segment among many pages and modalities. It’s the difference between a closed-book quiz on one chapter versus an open-book exam on the whole textbook — same question, very different cognitive demand.

• Explicit Reasoning Chains: Each QA pair comes with a step-by-step reasoning trace, not just a final answer. This matters for training because it gives the model a supervision signal on *how to reason, not just what to answer. It’s the difference between showing a student the answer and showing them the worked solution.

Framework Shift

Before (mainstream approach):        After (this paper):

  Paper                                Paper
    |                                    |
    v                                    v
  Annotate manually                  Extract segments
  (small, faithful, realistic)           |
    OR                                   v
  Auto-generate from snippets        Synthesize (Q,A,Chain)
  (large, unfaithful OR unrealistic)     |
                                         v
                                   Verify faithfulness
                                   (segment-level)
                                         |
                                         v
                                   Re-embed into full doc
                                   (document-level)
                                         |
                                         v
                                   Large, faithful, realistic

From choosing two out of three to getting all three via staged synthesis, the core shift is: decouple faithfulness (segment-level) from realism (document-level).

Expert Assessment

Problem choice: Real and well-motivated. The scale-faithfulness-realism trilemma is a genuine bottleneck for scientific document understanding. The timing is right — as foundation models get applied to scientific domains, the lack of large-scale, realistic training data becomes more visible.

Method maturity: The two-stage pipeline is solid engineering but not deeply novel. Claim-centric QA synthesis is a reasonable approach, and document-scale regrounding is a clean way to add difficulty. The key question is whether the regrounding actually tests the right skills (cross-modal navigation, long-range reasoning) or just adds noise.

Experimental integrity: The key experiments are: (1) does fine-tuning on SciMDR improve performance on held-out scientific QA benchmarks? (2) does the two-stage pipeline produce higher-quality data than single-stage alternatives? (3) how does performance scale with dataset size? All three need quantitative results. The paper would be stronger with human evaluation of data quality.

Writing quality: The abstract is clear and well-motivated. The methods section (not fully visible) likely needs more detail on the claim-centric synthesis process and the regrounding algorithm. The failure cases section is critical — when does this break? (Likely: papers with complex multi-figure reasoning, or highly technical domain-specific jargon.)

Verdict: weak accept — addresses a real problem with a reasonable two-stage approach, but the novelty is in the pipeline design rather than algorithmic innovation.

Takeaways

The two-stage synthesis pattern (generate at small scale for faithfulness, re-embed at large scale for realism) is broadly applicable. Whenever you’re synthesizing training data and face a quality-scale tradeoff, ask whether you can decouple the two stages.

The claim-centric QA approach (target specific verifiable claims rather than general summaries) is a clean way to ensure faithfulness. It’s worth trying in any domain where you need grounded QA pairs.

The explicit reasoning chain supervision (not just final answers) is important for training models that need to show their work. If your task requires interpretable reasoning, include reasoning chains in your training data.