ReContext: When LLMs Can See But Can't Use — Fixing the Last Mile of Long-Context Reasoning

Paper: 2607.02509 Authors: Yanjun Zhao, Ruizhong Qiu, Tianxin Wei, Yuanchen Bei, Zhining Liu, Lingjie Chen, Ismini Lourentzou, Hanghang Tong, Jingrui He Categories: cs.AI

The Gap

Here’s the frustrating situation we’ve hit in the long-context LLM world: models now advertise context windows of 128K, 200K, even millions of tokens. But when you actually test them on tasks requiring reasoning across that full context, they often fail — not because the evidence isn’t in the input, but because they can’t use it effectively. The gap between “context access” and “context utilization” is real and well-documented.

Prior attempts to fix this fall into three camps, each with clear limitations:

1. RAG / retrieval-augmented methods: Use an external retriever (BM25, embedding models) to find relevant chunks before feeding them to the LLM. Problem: the retriever and the reasoner have different notions of relevance, and you’re bottlenecked by the retriever’s understanding.

2. Context compression / pruning: Condense the long input into a shorter representation. Problem: you might throw away evidence that turns out to be critical, and compression is lossy by nature.

3. Fine-tuning for long contexts: Train the model specifically for long-input tasks. Problem: expensive, task-specific, and doesn’t generalize across backbones.

ReContext proposes a fourth path: use the model’s own attention patterns as a relevance signal, construct an evidence pool from that signal, and replay the most relevant evidence right before generation — all without any training.

[Long-context input with evidence scattered throughout]
        |
        v
[LLM produces initial attention patterns over full context]
        |
        v
[Assumption: attention signals encode genuine relevance]
        |
        v
[Extract query-conditioned evidence pool from attention]
        |
        v
[Recursively replay evidence pool before final answer]
        |
        v
[Model generates answer with evidence "pre-loaded"]
        |
        v
[Consistent improvement across 8 datasets, 3 backbones, 128K context]

The Increment

One sentence: Before this paper, if you wanted to improve a pre-trained LLM’s long-context reasoning, you either fine-tuned it or built external retrieval infrastructure; after this paper, you can get consistent gains by simply replaying the model’s own attention-sorted evidence at inference time, with zero training and zero external tools.

Core Mechanism

ReContext works in three phases, all happening at inference time. Let me walk through the data flow:

Phase 1 — Signal Extraction. You take the full long-context input (question + document) and run a single forward pass through the LLM. You don’t need the final answer — you only need the attention weights. These attention patterns tell you which parts of the context the model *thinks are relevant to the question. Specifically, you look at how much the question tokens attend to each segment of the context, giving you a relevance score per segment.

Phase 2 — Evidence Pool Construction. Using those relevance scores, you select the top-k most relevant segments and assemble them into an “evidence pool.” This pool is query-conditioned — different questions produce different evidence pools from the same document. The selection can be recursive: you can re-run the attention signal extraction on just the evidence pool to further refine it.

Phase 3 — Evidence Replay. You now construct the final input by placing the evidence pool *before the original full context. The model sees the condensed, relevant evidence first, then the full document, then the question. Because the evidence is replayed at the front, the model encounters it with fresh attention capacity — like putting the most important pages on top of the pile.

Phase 1: Signal Extraction
+---------------------------+
| Full Context + Question   |
|                           |
|   [doc segment 1]        |
|   [doc segment 2]        |
|   ...                     |
|   [doc segment N]        |
|   [question]              |
+---------------------------+
        |
        v  (forward pass, extract attention)
+---------------------------+
| Attention weight matrix   |
| Q_tokens x context_tokens |
+---------------------------+
        |
        v

Phase 2: Evidence Pool Construction
+---------------------------+
| Score each segment by     |
| attention from Q tokens   |
|                           |
| segment_2: 0.89           |
| segment_7: 0.76           |
| segment_1: 0.45           |
| ...                       |
+---------------------------+
        |
        v  (select top-k)
+---------------------------+
| Evidence Pool             |
| [segment_2][segment_7]..  |
+---------------------------+

Phase 3: Evidence Replay
+---------------------------+
| [Evidence Pool]           |  <-- replayed on top
| [Full Original Context]   |  <-- preserved in full
| [Question]                |
+---------------------------+
        |
        v  (final forward pass)
+---------------------------+
| Generated Answer          |
+---------------------------+

Structural metaphor — the trial lawyer’s preparation.

Imagine you’re a defense attorney the night before a major trial. You have a warehouse of documents — thousands of pages of depositions, contracts, emails, financial records. Your question: “Is my client innocent?”

Phase 1 is your gut scan. You flip through everything quickly, not reading deeply, just noticing which pages make you think “this matters.” You’re not a perfect judge of relevance, but your intuition — trained by years of practice — gives you a rough ranking. That’s the model’s attention: an imperfect but informative signal about what’s relevant.

Phase 2 is your evidence binder. You pull the top documents, organize them, maybe annotate them. You might even do a second pass on your binder — “wait, this email is actually only relevant because of the contract clause it references, let me make sure that’s flagged too.” That’s the recursive refinement of the evidence pool.

Phase 3 is how you set up your courtroom table. You put your key exhibits right in front of you, on top. The full warehouse of documents is still there — you can request any file at any time — but your most important evidence is *pre-positioned for immediate access. When the judge asks a question, your eyes go to the right pages first.

ReContext does exactly this for an LLM. The attention is the lawyer’s gut scan. The evidence pool is the organized binder. The replay is putting the binder on top of the table. The model still has access to the full context, but its most relevant evidence is pre-loaded at the position where it will be most naturally attended to.

Key Concepts

• Query-Conditioned Evidence Pool: This is the heart of ReContext. Instead of asking “what’s important in this document?” (which is context-dependent), it asks “what’s important *for this specific question?” The evidence pool changes depending on what you’re asking. Concretely: if you have a 128K document about a company and ask “What was Q3 revenue?”, the pool might contain financial tables and analyst commentary. If you ask “Who is the CEO?”, it contains the leadership section and board meeting notes. Same document, different pools. This is why they use the model’s own attention — it’s inherently question-dependent.

• Recursive Evidence Replay: The “recursive” part means you can apply the evidence extraction process multiple times. First pass: rough relevance signal over the full context. Second pass: refined signal over just the extracted evidence. Each iteration sharpens the signal-to-noise ratio. Think of it like iterative editing — your first draft picks up the main themes, your second draft cuts the fluff, your third draft finds the key sentences. The recursion is shallow (typically 1-2 rounds) but it meaningfully improves evidence quality over a single-pass approach.

• Associative Memory Framework: The paper’s theoretical lens, borrowed from cognitive science. In associative memory, you have a *store (your memories), a cue (what triggers recall), associations (strength of connection between cue and memory), and retrieval (pulling a memory into active consciousness). ReContext maps this: context = memory store, question = retrieval cue, attention weights = cue-trace associations, and replay = trace reactivation. This isn’t just metaphor decoration — it explains why replay works. In associative memory theory, replaying a trace strengthens its accessibility. By replaying evidence in the prompt, you’re essentially strengthening the model’s ability to attend to it during generation.

Framework Shift

Before (mainstream approach):        After (this paper):

[External Retriever]                 [Model's Own Attention]
       |                                    |
       v                                    v
[Top-k chunks via BM25/embeddings]   [Top-k segments via attention scores]
       |                                    |
       v                                    v
[Feed chunks to LLM]                [Replay evidence before full context]
       |                                    |
       v                                    v
[Answer]                             [Answer]
       |                                    |
       v                                    v
[Disagreement between retriever      [No external dependency;
 and model about what's relevant]     model's own notion of relevance]

From external retrieval to self-retrieval, the core shift is: the model already knows what’s relevant — it just needs a second chance to look at it.

Expert Assessment

Problem choice: This is a genuine and well-motivated gap. The “lost in the middle” phenomenon and related findings are well-established, and the access-vs-utilization distinction is crisp and practically important. It sits squarely in the hot zone of long-context LLM research and addresses a real pain point for practitioners deploying these models. That said, the problem space is getting crowded — there are many papers trying to make long contexts work better, and this paper doesn’t deeply engage with some recent competitors (e.g., context caching techniques, positional encoding improvements).

Method maturity: The core insight — use attention as a relevance signal and replay evidence — is elegant and genuinely clever. It’s not brute force at all; it’s a surgical intervention that requires no training and adds minimal overhead. The recursive refinement is a nice touch. However, I’m curious about edge cases: what happens when the attention signal itself is unreliable (which is known to happen in some architectures)? The paper doesn’t deeply address failure modes. Also, the method requires a full additional forward pass for signal extraction, which could be significant at 128K context lengths — the computational cost analysis is thin.

Experimental integrity: Eight datasets across three backbones (Qwen3-4B, Qwen3-8B, Llama3-8B) is decent breadth. The “best average rank” claim is reasonable. However, I’d like to see: (1) ablation on the number of recursion steps — the paper mentions it but could be more thorough; (2) comparison against simpler baselines like “just move the question to the end” or “just randomly sample k segments as replay”; (3) wall-clock runtime comparisons, not just accuracy. The baselines are fair but not exhaustive. No major red flags, but some yellow flags on computational cost transparency.

Writing quality: The abstract and introduction are strong — the gap is clearly articulated. The theoretical section (associative memory framework) is the weakest part; it feels like it’s trying to justify the method post-hoc rather than motivating it. If the authors had *started from the associative memory theory and derived the method, the paper would be significantly stronger. As written, the theory reads like an afterthought dressed up in cognitive science language. The experiments section is solid but could be more visually engaging — some attention heatmap visualizations of the evidence selection would go a long way.

Verdict: weak accept — The core idea is clean, the execution is competent, and the results are consistent. The main weaknesses are incomplete ablations and a theoretical section that doesn’t quite land. This is the kind of paper that will get cited by people trying to improve their long-context pipelines, which is a good sign.

Takeaways

Three concrete things a practitioner can steal:

1. The “attention-as-retriever” pattern: Don’t build an external retrieval system when the model already has relevance signals built in. Run a throwaway forward pass, extract attention weights over segments, use those as a retrieval score. This works with any transformer-based model and requires zero additional infrastructure.

2. Replay as a prompt engineering technique: Even without the full ReContext pipeline, the insight that *putting relevant evidence at the beginning of the context improves utilization is actionable. If you know which parts of a long document are relevant, prepend them. The positional bias in transformers works for you here.

3. Recursive refinement on your own signals: If you have a noisy signal (like attention weights), you can improve it by running the signal extraction multiple times on successively refined inputs. This “self-distillation” pattern is cheap and doesn’t require any training — it’s worth trying whenever you have a noisy selection process that could benefit from iteration.