OfficeQA Pro: Why Frontier AI Still Fails at Enterprise Document Reasoning

Paper: 2603.08655 Authors: Krista Opsahl-Ong, Arnav Singhvi, Jasmine Collins, Ivan Zhou, Cindy Wang, Ashutosh Baheti, Owen Oertell, Jacob Portes, Sam Havens, Erich Elsen Categories: cs.AI, cs.CL, cs.IR

The Gap

Existing document QA benchmarks like SQuAD, Natural Questions, and HotpotQA test models on clean, curated datasets with single documents or simple multi-hop reasoning. Enterprise reality is messier: 89,000-page corpora spanning decades, heterogeneous formats mixing prose and tables, questions requiring precise numerical extraction across multiple documents. Current benchmarks don’t capture this complexity, leaving a blind spot between impressive lab results and disappointing production deployments.

Problem: Lab benchmarks != Enterprise reality
   |
   v
Assumption: Real enterprise tasks need multi-document
            reasoning over heterogeneous data at scale
   |
   v
Method: Build benchmark from 100 years of Treasury
        Bulletins (89K pages, 26M numbers, 133 questions)
   |
   v
Evidence: Frontier models <35% accuracy even with
          full corpus access (vs >90% on lab benchmarks)
   |
   v
Conclusion: Significant gap remains for enterprise-grade
            grounded reasoning

The Increment

One sentence: Before this paper, we thought frontier LLMs were close to production-ready for document reasoning; after, we know they fail on over 65% of realistic enterprise tasks even with perfect document access.

Core Mechanism

OfficeQA Pro consists of three components: a corpus, a question set, and an evaluation protocol. The corpus is 89,000 pages of U.S. Treasury Bulletins from 1927-2024, containing both unstructured text and dense tabular data with 26 million numerical values. The question set comprises 133 carefully designed questions that require locating relevant documents, parsing both text and tables, and performing analytical reasoning across multiple sources.

The evaluation protocol tests models under three access conditions: parametric knowledge only (no external data), web access (simulating RAG with search), and direct corpus access (oracle retrieval). For each condition, the paper measures exact match accuracy and conducts ablations on document representation (raw vs structured parsing), retrieval strategy, and model selection. The structured representation uses Databricks’ ai_parse_document to convert PDFs into a format that preserves table structure and spatial relationships.

Think of this like stress-testing bridges. Lab benchmarks are like testing toy bridges in controlled conditions—clean materials, perfect weather, known loads. OfficeQA Pro is like testing real bridges under actual conditions: decades of weathering (document age), mixed materials (text + tables), unpredictable traffic patterns (diverse question types), and real-world constraints (must find right documents first). The corpus is the harsh environment, the questions are the stress tests, and the results show which bridge designs (models + methods) actually hold up when it matters. Just as a toy bridge that works in the lab might collapse under real traffic, models that ace SQuAD might fail when faced with 89,000 pages of Treasury data.

Key Concepts

• Grounded reasoning: This means the model must derive answers from specific documents rather than relying on memorized knowledge. Imagine a lawyer who must cite exact page numbers and quotes from case law rather than arguing from general legal principles. The model can’t just “know” that Treasury debt increased in 2008—it must locate the specific bulletin, parse the relevant table, extract the numbers, and compute the answer. This is harder because it requires precise document understanding, not just plausible generation.

• Heterogeneous document corpus: Real enterprise data isn’t uniform. A single Treasury Bulletin might contain narrative text explaining policy changes, dense tables with hundreds of numerical entries, footnotes with caveats, and charts. The model must handle all these formats simultaneously and understand how they relate. It’s like asking someone to cook a meal using ingredients from different cuisines, each requiring different preparation techniques—you can’t just blend everything together.

• Multi-document reasoning: Many questions require synthesizing information across multiple documents. For example: “How did the debt-to-GDP ratio change between 1950 and 1980?” requires finding debt figures from bulletins across 30 years, GDP data from potentially different sources, computing ratios, and tracking changes. This is fundamentally different from single-document QA where all information is localized.

Framework Shift

Before (lab benchmarks):              After (OfficeQA Pro):
                                      
  Question                              Question
     |                                     |
     v                                     v
  [Clean doc]                         [89K pages]
     |                                  /  |  \
  Extract                          Find  Parse  Reason
     |                              /      |      \
  Answer                       [Text] [Tables] [Multi-doc]
                                    \      |      /
  Accuracy: ~90%                     Synthesize
                                         |
                                      Answer
                                         
                                    Accuracy: ~34%

From testing reading comprehension on curated paragraphs to testing end-to-end information work on realistic corpora, the core shift is from isolated skill evaluation to integrated task performance.

Expert Assessment

Problem choice: This is a genuine gap. The disconnect between benchmark performance and production failures is a real pain point for enterprises deploying LLMs. The choice of Treasury Bulletins is smart—publicly available, naturally complex, spans decades, and represents actual enterprise use cases (financial analysis, compliance, reporting).

Method maturity: As a benchmark paper, the “method” is corpus and question design. The corpus is well-chosen but questions could be more diverse—133 questions is relatively small, and we don’t know if they cover the full distribution of enterprise reasoning tasks. The ablations are informative (structured parsing helps, retrieval matters), but the paper doesn’t deeply investigate why models fail or what specific capabilities are missing.

Experimental integrity: Baselines are fair and comprehensive (multiple frontier models, various access conditions). The numbers are credible—<5% parametric, <12% with web, ~34% with corpus access tells a consistent story of increasing but still insufficient performance. One concern: we don’t see inter-annotator agreement on answer correctness, which matters for numerical questions where “close enough” might be debatable.

Writing quality: The paper is clear and well-structured, but the analysis section feels thin. The authors show that structured parsing helps (+16.1% relative gain) but don’t dig into what aspects of structure matter most. The error analysis is minimal—we don’t get a taxonomy of failure modes or insights into what makes questions hard. Rewriting Section 5 (Analysis) with deeper investigation of failure patterns would elevate the paper significantly.

Verdict: weak accept — Useful benchmark that exposes a real gap, but limited analytical depth and relatively small question set prevent it from being a strong contribution.

Takeaways

For practitioners deploying document QA systems: Don’t trust lab benchmark numbers. Test on your actual corpus with realistic questions. The 16.1% gain from structured document parsing (ai_parse_document) suggests investing in document preprocessing is worthwhile—convert PDFs to structured formats that preserve tables and spatial relationships before feeding to LLMs.

For researchers: There’s a massive opportunity in multi-document reasoning over heterogeneous data. The gap between 34% and human performance is where the next generation of methods will be built. Focus on table understanding, cross-document synthesis, and retrieval over large corpora.

For benchmark designers: Scale and realism matter. Small, clean datasets create a false sense of progress. Build benchmarks from real enterprise data, even if it’s messier and harder to annotate.

Specific technique to steal: The evaluation protocol of testing under three access conditions (parametric, web, corpus) is a clean way to isolate different failure modes. Use this pattern when evaluating your own systems to understand where performance bottlenecks lie.