Document Reconstruction Unlocks Scalable Long-Context RLVR

Paper: 2602.08237 Authors: Yao Xiao, Lei Wang, Yue Deng, Guanzheng Chen, Ziqi Jin, Jung-jae Kim, Xiaoli Li, Roy Ka-wei Lee, Lidong Bing Categories: cs.CL, cs.LG

Abstract

This paper presents an unsupervised method to improve language model long-context abilities without requiring human annotations. The approach replaces paragraphs in long documents with placeholders, then trains the model via reinforcement learning to reconstruct the original document by identifying and correctly ordering the missing sections. The method is evaluated on RULER and LongBench v2, showing consistent improvements. The authors release code, data, and models.

Key Contributions

• Unsupervised long-context training: No human annotations or task-specific labels required
• Document reconstruction as RL reward: A naturally verifiable objective for long-context reasoning
• Scalable data generation: Any long document can be converted into a training instance
• Open release: Code, data, and trained models publicly available

Motivation

Training LLMs to handle long contexts effectively remains challenging. Existing approaches typically rely on:

• Supervised fine-tuning: Requires expensive human-annotated long-context QA pairs
• Synthetic data: Often limited in diversity and naturalness
• Continued pretraining: Improves context window but not necessarily reasoning over long contexts

Reinforcement Learning with Verifiable Rewards (RLVR) has shown promise for improving reasoning, but applying it to long-context tasks has been limited by the difficulty of constructing verifiable long-context tasks at scale.

Method

Document Reconstruction Task

Given a long document $D = [p_1, p_2, \ldots, p_n]$ consisting of $n$ paragraphs:

1. Select paragraphs: Randomly choose $k$ paragraphs $\{p_{i_1}, p_{i_2}, \ldots, p_{i_k}\}$ for removal
2. Create placeholders: Replace each selected paragraph with a placeholder token $[\text{BLANK}_j]$
3. Shuffle removed paragraphs: Present the removed paragraphs in random order as candidates
4. Task: The model must assign each candidate paragraph to its correct placeholder position

The corrupted document with candidates forms the input:

$\text{Input} = [D_{\text{corrupted}}, \text{Candidates}: \{c_1, c_2, \ldots, c_k\}]$

The model must output the correct mapping: $[\text{BLANK}_j] \rightarrow c_{\sigma(j)}$ where $\sigma$ is the correct permutation.

Why This Task Works for Long-Context Training

The reconstruction task requires multiple long-context skills simultaneously:

• Global comprehension: Understanding the document’s overall structure and flow
• Local coherence: Matching paragraph boundaries based on topical and stylistic continuity
• Positional reasoning: Determining the correct order among multiple candidates
• Long-range dependency: Connecting information across distant parts of the document

Reinforcement Learning with Verifiable Rewards

The reconstruction task provides a naturally verifiable reward signal:

$R = \frac{1}{k}\sum_{j=1}^{k} \mathbb{1}[\text{predicted}(j) = \text{correct}(j)]$

This reward is:

• Exact: No ambiguity in correctness (either the paragraph is correctly placed or not)
• Granular: Partial credit for partially correct reconstructions
• Scalable: Automatically computed without human judgment

The RL training uses this reward to optimize the model’s policy for long-context reasoning:

$\mathcal{L}_{\text{RL}} = -\mathbb{E}_{\pi_\theta}[R \cdot \log \pi_\theta(a|s)]$

Data Generation Pipeline

The data generation is fully automatic:

1. Collect long documents from diverse sources (books, articles, reports)
2. Filter for quality and minimum length
3. Segment into paragraphs using natural boundaries
4. Generate multiple reconstruction instances per document by varying which paragraphs are removed
5. Control difficulty by adjusting the number of removed paragraphs $k$

This pipeline can generate virtually unlimited training data from any text corpus.

Experimental Results

RULER Benchmark

RULER evaluates long-context capabilities across multiple tasks:

• Retrieval: Finding specific information in long contexts
• Multi-hop reasoning: Connecting information across document sections
• Aggregation: Summarizing information distributed throughout the context

Results show consistent improvements across all RULER subtasks, with particularly strong gains on multi-hop reasoning tasks that require connecting distant information.

LongBench v2

On the more challenging LongBench v2 benchmark:

• Improvements across diverse real-world long-context tasks
• Strong performance on tasks requiring document understanding
• Gains maintained across different context lengths

Scaling Properties

The method shows favorable scaling behavior:

• More reconstruction instances: Performance improves with more training data (easy to generate)
• Harder instances: Increasing the number of removed paragraphs $k$ provides curriculum-like difficulty scaling
• Longer documents: Training on longer documents transfers to improved performance at all context lengths

Analysis

What the Model Learns

Analysis of the trained models reveals improved capabilities in:

• Structural understanding: Better modeling of document organization and flow
• Coherence detection: Improved ability to detect topical transitions
• Position sensitivity: Enhanced awareness of where information appears in long contexts
• Information integration: Better at combining information from multiple document locations

Comparison with Supervised Approaches

Despite being fully unsupervised, the document reconstruction approach:

• Matches or approaches supervised long-context fine-tuning on most benchmarks
• Significantly outperforms continued pretraining alone
• Provides more diverse training signal than task-specific supervised data

Difficulty Analysis

The number of removed paragraphs $k$ controls task difficulty:

• $k = 2$: Relatively easy, model can use simple local coherence cues
• $k = 4\text{-}6$: Moderate difficulty, requires global document understanding
• $k = 8+$: Challenging, requires sophisticated reasoning about document structure

Training with a curriculum from easy to hard instances yields the best results.

Significance

This work makes a compelling case for document reconstruction as a scalable, unsupervised objective for long-context RL training. The key insight is that the structure of natural documents provides a rich, freely available source of verifiable rewards. By leveraging this signal through RL, the method improves long-context reasoning without the bottleneck of human annotation. The open release of code, data, and models makes this approach immediately accessible to the research community.