Paper: 2602.12247 Authors: Nick Ferguson, Josh Pennington, Narek Beghian, Aravind Mohan, Douwe Kiela, Sheshansh Agrawal, Thien Hang Nguyen Categories: cs.LG, cs.AI

Abstract

ExtractBench addresses a critical gap in evaluating LLM-based structured extraction from unstructured documents. While enterprises increasingly rely on LLMs to convert PDFs into standardized JSON formats, no comprehensive benchmark has existed to measure performance on realistic, complex schemas. This paper introduces both a benchmark dataset of 35 PDF documents paired with human-annotated gold labels across 12,867 evaluatable fields, and a principled evaluation methodology that treats JSON schemas as executable specifications. Each field declares its own scoring metric, accommodating the semantic nuances of nested extraction—exact matching for identifiers, tolerance for numerical quantities, and semantic equivalence for text fields. Baseline evaluations of frontier models (GPT-5/5.2, Gemini-3 Flash/Pro, Claude 4.5 Opus/Sonnet) reveal alarming unreliability: performance degrades sharply with schema breadth, reaching 0% valid output on a 369-field financial reporting schema across all tested models.

Key Contributions

  • First enterprise-scale benchmark: 35 PDF documents spanning economically valuable domains (financial reports, invoices, contracts, medical records) with schema complexities ranging from tens to hundreds of fields
  • Principled evaluation framework: Schema-driven methodology where each field specifies its scoring metric, enabling nuanced assessment of nested structures, array alignment, and distinguishing omissions from hallucinations
  • Comprehensive baseline analysis: Systematic evaluation of frontier LLMs revealing severe performance degradation with schema complexity
  • Open-source release: Complete benchmark, evaluation code, and baseline results available at https://github.com/ContextualAI/extract-bench

The Structured Extraction Challenge

Structured extraction from unstructured documents represents a fundamental bottleneck in enterprise data pipelines. PDFs, the de facto standard for document exchange, contain rich information locked in visual layouts, tables, and free-form text. Downstream systems—databases, analytics platforms, compliance tools—require this data in standardized, machine-readable formats like JSON.

Traditional approaches relied on brittle rule-based systems or specialized OCR pipelines. LLMs promised a paradigm shift: general-purpose models that could understand document semantics and extract structured data through natural language instructions. However, the gap between promise and reality remains poorly quantified.

The challenge extends beyond simple field extraction. Enterprise schemas exhibit several complexities:

  1. Nested hierarchies: Documents contain objects within objects (e.g., a contract with multiple parties, each having addresses and contact information)
  2. Variable-length arrays: Invoices list arbitrary numbers of line items; financial reports contain multiple transactions
  3. Heterogeneous field types: Identifiers demand exact matches, quantities tolerate rounding errors, names accept semantic equivalents
  4. Schema breadth: Real-world schemas span hundreds of fields, testing model capacity and attention

No existing benchmark captures this complexity end-to-end, from PDF input to validated JSON output.

Methodology and Benchmark Design

ExtractBench’s design philosophy treats the JSON schema as an executable specification. Rather than imposing a single scoring metric across all fields, the framework allows each field to declare its correctness criterion:

  • Exact match: For identifiers, codes, and categorical values where precision is non-negotiable
  • Numerical tolerance: For quantities where ∣predicted−actual∣<ϵ|predicted - actual| < \epsilon suffices
  • Semantic equivalence: For text fields where “John Smith” and “J. Smith” should match
  • Structural alignment: For arrays where element correspondence must be established before scoring

The benchmark construction involved three phases:

  1. Document selection: 35 PDFs chosen to represent diverse layouts, domains, and extraction challenges. Documents include multi-page financial statements, dense legal contracts, tabular invoices, and semi-structured medical records.

  2. Schema definition: JSON schemas ranging from 20 to 369 fields, designed to reflect real enterprise requirements. Schemas include nested objects (depth up to 4 levels), arrays of varying cardinality, and mixed field types.

  3. Gold label annotation: Human experts manually extracted ground truth for all 12,867 fields. Annotators followed detailed guidelines to ensure consistency, with inter-annotator agreement measured on a subset.

The evaluation framework implements schema-aware scoring:

1 & \text{if } \text{metric}(predicted, gold) = \text{true} \\ 0 & \text{otherwise} \end{cases}$$ $$\text{Document Score} = \frac{\sum \text{Field Scores}}{\text{Total Fields}}$$ For arrays, the framework first solves an assignment problem to align predicted and gold elements, then scores matched pairs. Unmatched predictions count as hallucinations; unmatched gold elements count as omissions. ## Experimental Results and Analysis The baseline evaluation tested six frontier models across all 35 documents. Results reveal a stark reality: current LLMs are not production-ready for complex structured extraction. **Performance vs. Schema Complexity**: A clear inverse relationship emerges. On schemas with fewer than 50 fields, models achieve 60-80% field-level accuracy. Performance drops to 30-50% for schemas with 100-200 fields. On the 369-field financial reporting schema, all models produced 0% valid output—failing even basic JSON syntax validation. **Model Comparison**: GPT-5.2 and Claude 4.5 Opus lead on simpler schemas but degrade similarly on complex ones. Gemini-3 Flash, optimized for speed, trails by 10-15 percentage points across all complexity levels. No model demonstrates robust scaling to enterprise-scale schemas. **Error Analysis**: Manual inspection of failures reveals several patterns: - **Structural hallucinations**: Models invent plausible-sounding but incorrect nested structures - **Array misalignment**: Line items get shuffled or merged incorrectly - **Attention failures**: Fields late in long schemas are frequently omitted or filled with generic placeholders - **Type confusion**: Numerical fields populated with text descriptions, categorical fields with free-form text **Domain Variation**: Financial documents prove most challenging (average 35% accuracy), likely due to dense tabular layouts and domain-specific terminology. Invoices perform best (55% accuracy) thanks to standardized formats. Medical records fall in between (45% accuracy), with models struggling on handwritten annotations and non-standard abbreviations. ## Implications for Production Deployment The results carry significant implications for enterprises deploying LLM-based extraction: **Reliability gap**: 0% success on realistic schemas means current models cannot replace human review for mission-critical applications. Financial reporting, regulatory compliance, and legal document processing remain human-dependent. **Schema design constraints**: Organizations must either simplify schemas (sacrificing information richness) or implement extensive post-processing and validation layers (negating automation benefits). **Hybrid architectures**: The path forward likely involves decomposing complex extraction into subtasks—specialized models for tables, another for entity recognition, rule-based validation—rather than end-to-end LLM solutions. **Evaluation necessity**: ExtractBench demonstrates that generic benchmarks (GLUE, SuperGLUE) and even document-focused ones (DocVQA) fail to predict performance on structured extraction. Task-specific evaluation is essential. ## Future Directions ExtractBench opens several research avenues: 1. **Schema-aware pretraining**: Models explicitly trained to respect JSON schema constraints during generation 2. **Iterative refinement**: Multi-pass extraction where models self-verify and correct outputs 3. **Retrieval-augmented extraction**: Incorporating schema documentation and example extractions as context 4. **Specialized architectures**: Models with explicit structural inductive biases for nested data The benchmark will expand with additional domains (scientific papers, government forms, e-commerce listings) and multilingual documents. The evaluation framework will incorporate confidence calibration metrics, measuring whether models know when they're uncertain. ## Takeaways 1. Frontier LLMs achieve 0% valid output on enterprise-scale schemas (369 fields), revealing a critical reliability gap for production deployment 2. Performance degrades sharply with schema complexity, dropping from 60-80% accuracy on simple schemas to 30-50% on moderately complex ones 3. Current models exhibit systematic failures: structural hallucinations, array misalignment, attention failures on long schemas, and type confusion 4. ExtractBench provides the first principled evaluation methodology for nested structured extraction, treating schemas as executable specifications with field-specific scoring metrics 5. The path to production-ready extraction likely requires hybrid architectures combining specialized models, rule-based validation, and human oversight rather than end-to-end LLM solutions

论文: 2602.12247 作者: Nick Ferguson, Josh Pennington, Narek Beghian, Aravind Mohan, Douwe Kiela, Sheshansh Agrawal, Thien Hang Nguyen 分类: cs.LG, cs.AI

摘要

ExtractBench解决了评估基于大语言模型的非结构化文档结构化提取中的关键空白。虽然企业越来越依赖大语言模型将PDF转换为标准化JSON格式,但此前并不存在全面的基准来衡量在真实复杂模式上的性能。本文引入了一个基准数据集,包含35个PDF文档,配有人工标注的黄金标签,涵盖12,867个可评估字段,以及一个原则性评估方法,将JSON模式视为可执行规范。每个字段声明自己的评分指标,适应嵌套提取的语义细微差别——标识符的精确匹配、数值量的容差以及文本字段的语义等价。对前沿模型(GPT-5/5.2、Gemini-3 Flash/Pro、Claude 4.5 Opus/Sonnet)的基线评估揭示了令人担忧的不可靠性：性能随模式广度急剧下降,在369字段的财务报告模式上,所有测试模型的有效输出率均为0%。

主要贡献

  • 首个企业级基准: 35个PDF文档涵盖具有经济价值的领域(财务报告、发票、合同、医疗记录),模式复杂度从数十个字段到数百个字段不等
  • 原则性评估框架: 模式驱动的方法,每个字段指定其评分指标,实现对嵌套结构、数组对齐的细致评估,并区分遗漏与幻觉
  • 全面的基线分析: 对前沿大语言模型的系统评估,揭示了随模式复杂度增加的严重性能退化
  • 开源发布: 完整的基准、评估代码和基线结果可在 https://github.com/ContextualAI/extract-bench 获取

结构化提取挑战

从非结构化文档中进行结构化提取代表了企业数据管道中的基本瓶颈。PDF作为文档交换的事实标准,包含锁定在视觉布局、表格和自由文本中的丰富信息。下游系统——数据库、分析平台、合规工具——需要这些数据以标准化、机器可读的格式(如JSON)呈现。

传统方法依赖于脆弱的基于规则的系统或专门的OCR管道。大语言模型承诺了范式转变：通用模型可以理解文档语义并通过自然语言指令提取结构化数据。然而,承诺与现实之间的差距仍然缺乏量化。

挑战不仅限于简单的字段提取。企业模式表现出几种复杂性：

  1. 嵌套层次结构: 文档包含对象中的对象(例如,包含多个当事方的合同,每个当事方都有地址和联系信息)
  2. 可变长度数组: 发票列出任意数量的行项目;财务报告包含多个交易
  3. 异构字段类型: 标识符要求精确匹配,数量容忍舍入误差,名称接受语义等价
  4. 模式广度: 真实世界的模式跨越数百个字段,测试模型容量和注意力

现有基准都没有端到端地捕捉这种复杂性,从PDF输入到验证的JSON输出。

方法论与基准设计

ExtractBench的设计理念将JSON模式视为可执行规范。该框架不是在所有字段上强加单一评分指标,而是允许每个字段声明其正确性标准：

  • 精确匹配: 用于标识符、代码和分类值,其中精度不可妥协
  • 数值容差: 用于数量,其中 ∣predicted−actual∣<ϵ|predicted - actual| < \epsilon 即可
  • 语义等价: 用于文本字段,其中”John Smith”和”J. Smith”应该匹配
  • 结构对齐: 用于数组,其中必须在评分前建立元素对应关系

基准构建涉及三个阶段：

  1. 文档选择: 选择35个PDF以代表不同的布局、领域和提取挑战。文档包括多页财务报表、密集的法律合同、表格发票和半结构化医疗记录。

  2. 模式定义: JSON模式范围从20到369个字段,旨在反映真实的企业需求。模式包括嵌套对象(深度达4层)、不同基数的数组以及混合字段类型。

  3. 黄金标签标注: 人类专家手动提取所有12,867个字段的真实值。标注者遵循详细指南以确保一致性,并在子集上测量标注者间一致性。

评估框架实现模式感知评分：

1 & \text{如果 } \text{metric}(predicted, gold) = \text{true} \\ 0 & \text{否则} \end{cases}$$ $$\text{文档得分} = \frac{\sum \text{字段得分}}{\text{总字段数}}$$ 对于数组,框架首先解决分配问题以对齐预测和黄金元素,然后对匹配的配对进行评分。未匹配的预测计为幻觉;未匹配的黄金元素计为遗漏。 ## 实验结果与分析 基线评估在所有35个文档上测试了六个前沿模型。结果揭示了一个严峻的现实：当前的大语言模型尚未为复杂结构化提取做好生产准备。 **性能与模式复杂度**: 出现了明显的反比关系。在少于50个字段的模式上,模型达到60-80%的字段级准确率。对于100-200个字段的模式,性能下降到30-50%。在369字段的财务报告模式上,所有模型产生0%的有效输出——甚至无法通过基本的JSON语法验证。 **模型比较**: GPT-5.2和Claude 4.5 Opus在较简单的模式上领先,但在复杂模式上同样退化。针对速度优化的Gemini-3 Flash在所有复杂度级别上落后10-15个百分点。没有模型展示出对企业级模式的稳健扩展能力。 **错误分析**: 对失败的人工检查揭示了几种模式： - **结构性幻觉**: 模型发明听起来合理但不正确的嵌套结构 - **数组错位**: 行项目被错误地打乱或合并 - **注意力失败**: 长模式中后期的字段经常被省略或填充通用占位符 - **类型混淆**: 数值字段填充文本描述,分类字段填充自由文本 **领域变化**: 财务文档最具挑战性(平均35%准确率),可能是由于密集的表格布局和特定领域术语。发票表现最好(55%准确率),得益于标准化格式。医疗记录介于两者之间(45%准确率),模型在手写注释和非标准缩写上遇到困难。 ## 生产部署的影响 结果对部署基于大语言模型的提取的企业具有重要意义： **可靠性差距**: 在真实模式上0%的成功率意味着当前模型无法取代关键任务应用中的人工审查。财务报告、监管合规和法律文档处理仍然依赖人工。 **模式设计约束**: 组织必须简化模式(牺牲信息丰富性)或实施广泛的后处理和验证层(抵消自动化优势)。 **混合架构**: 前进的道路可能涉及将复杂提取分解为子任务——专门用于表格的模型、用于实体识别的另一个模型、基于规则的验证——而不是端到端的大语言模型解决方案。 **评估必要性**: ExtractBench证明通用基准(GLUE、SuperGLUE)甚至面向文档的基准(DocVQA)都无法预测结构化提取的性能。特定任务的评估至关重要。 ## 未来方向 ExtractBench开辟了几个研究方向： 1. **模式感知预训练**: 明确训练模型在生成过程中遵守JSON模式约束 2. **迭代细化**: 多遍提取,模型自我验证和纠正输出 3. **检索增强提取**: 将模式文档和示例提取作为上下文纳入 4. **专门架构**: 具有嵌套数据显式结构归纳偏置的模型 基准将扩展到其他领域(科学论文、政府表格、电子商务列表)和多语言文档。评估框架将纳入置信度校准指标,衡量模型是否知道何时不确定。 ## 要点总结 1. 前沿大语言模型在企业级模式(369个字段)上达到0%的有效输出,揭示了生产部署的关键可靠性差距 2. 性能随模式复杂度急剧下降,从简单模式上的60-80%准确率下降到中等复杂模式上的30-50% 3. 当前模型表现出系统性失败：结构性幻觉、数组错位、长模式上的注意力失败以及类型混淆 4. ExtractBench为嵌套结构化提取提供了首个原则性评估方法,将模式视为具有字段特定评分指标的可执行规范 5. 实现生产就绪提取的路径可能需要混合架构,结合专门模型、基于规则的验证和人工监督,而不是端到端的大语言模型解决方案