Reading Is Not Using: Long Contexts Break the Link Between Retrieval and Judgment

Paper: 2608.24842 Authors: Miao Liu, Zhizhe Liu Categories: cs.AI, cs.CL

The Gap

LLMs are being deployed as AI analysts that read financial disclosures and support investment decisions. The way these systems are evaluated is straightforward: measure what they can retrieve. Did the model find the risk disclosure? Did it quote the right figure? Retrieval accuracy is measurable, comparable, and cheap.

But retrieval is not the deliverable. The deliverable is a judgment that is informed by the disclosure — and those are different claims. A system can retrieve a document perfectly and then form the same opinion it would have formed without it. When that happens, the evaluation still passes, because every step it checks went right. The paper names this the retrieval-integration gap in long-context financial analysis, and it is a gap between two things that are routinely conflated: having access to information, and being influenced by it.

The measurement problem is what makes this hard to notice. If you test with a short context, retrieval and integration tend to co-occur, so retrieval accuracy looks like a reasonable proxy. The gap opens up as the context grows, and it opens in a way that retrieval metrics structurally cannot detect.

   WHAT IS EVALUATED vs WHAT IS DELIVERED

   evaluated:   "can the model RETRIEVE the disclosure?"
                -> measurable, comparable, cheap
                     |
   delivered:   "is the model's JUDGMENT INFLUENCED by it?"
                -> the actual product
                     |
        +------------+------------+
        v                         v
   retrieve + be influenced   retrieve + ignore
   (goal)                     (silent failure)
        |                         |
        +------------+------------+
                     v
   BOTH PASS a retrieval-based evaluation
        |
        v
   [GAP] retrieval-integration gap:
         access to information is not influence by it

The Increment

One sentence: Before this paper, an AI analyst could be certified by what it retrieved; after it, holding the focal information fixed while swelling unrelated context shows that a disclosure’s influence on judgments collapses to the noise floor even though retrieval stays accurate — and that workflow architecture decides whether it collapses at all.

Core Mechanism

The design isolates the variable cleanly, which is what makes the finding interpretable.

The focal information is held fixed; only unrelated context varies. A risk disclosure about the company under analysis stays the same. What changes is the amount of irrelevant material surrounding it, swept from 2,000 to 128,000 tokens. This is the right control: if the disclosure is identical, any change in its influence is attributable to context length, not to the content of the disclosure.

Influence is measured, not retrieval. The outcome is whether the disclosure moves the investment judgment. And the result is stark: as unrelated context grows, the disclosure’s influence falls to the experimental noise floor even as direct retrieval remains accurate. The model still finds it. The model stops being moved by it. The two measurements diverge in opposite directions, which is exactly what a retrieval-only evaluation cannot see.

Three further results broaden the claim:

The pattern replicates across model families and judgment tasks, and — the strongest form of the evidence — in experiments removing real disclosures from actual 10-K filings. Using real filings rather than synthetic probes means the effect is not an artifact of contrived inputs.

More capable models postpone but do not eliminate the gap. Scaling helps, and only in the sense of raising the context length at which the collapse begins. This matters for anyone treating capability as a solution: the gap is not a deficiency that a bigger model closes.

Causal memory interventions identify the transmission channel. Compressed summaries and source-text lookup jointly transmit disclosures into judgments. The word “jointly” is the finding: neither mechanism alone carries the influence, which means the pathway has two components and improving just one will not restore it.

Workflow architecture decides whether transmission succeeds. This is the constructive result. Chunk-and-summarize pipelines evict relevant information — the standard way to handle long documents is the very thing that breaks integration. By contrast, a targeted, structured restatement adjacent to the decision restores the disclosure’s influence. Not more retrieval, not a longer context: a restatement, placed where the decision is being made.

   DESIGN AND FINDINGS

   focal-firm disclosure : HELD FIXED
   unrelated context     : 2,000 -> 128,000 tokens
        |
        v
   MEASURE TWO THINGS SEPARATELY
     [retrieval accuracy]  -> stays accurate
     [judgment influence]  -> falls to NOISE FLOOR
        |
        v
   replicates across model families, judgment tasks,
   and real 10-K disclosure-removal experiments
        |
        v
   more capable models: POSTPONE the collapse,
                        do not eliminate it
        |
        v
   TRANSMISSION CHANNEL (causal memory interventions)
     compressed summaries + source-text lookup
     -> JOINTLY carry the influence (neither alone)
        |
        v
   WORKFLOW ARCHITECTURE DECIDES IT
     chunk-and-summarize   -> EVICTS relevant info
     structured restatement adjacent to the decision
                           -> RESTORES influence

Think of it as a briefing delivered to someone reading a thousand-page document. Hand them the page with the critical risk warning, and they see it. Hand them the same page buried inside a thousand pages of unrelated filings, and they still pass a comprehension check on that page — they can quote it — but their actual decision is made from the general impression of the whole stack. The warning was read and not absorbed, and no test of reading comprehension would catch it. The paper’s constructive finding says the fix is not to shorten the stack or to hand over the page more forcefully; it is to restate the warning next to the decision, so the act of deciding and the content of the warning are adjacent in the reasoning.

Key Concepts

• The retrieval-integration gap: the divergence between having access to information and being influenced by it, which grows with context length and is invisible to retrieval metrics. It matters because the metrics in common use measure the wrong end of the pipeline.
• Influence falling to the noise floor: the specific measurement. A disclosure whose effect on judgments is indistinguishable from noise is functionally absent, regardless of how accurately the system can produce it on request.
• Joint transmission channel: compressed summaries and source-text lookup carry the influence together, neither alone. It is an architectural finding with an immediate implication — optimising one component of a two-part channel does not restore the signal.
• Evicting versus restating: chunk-and-summarize pipelines evict the relevant content, while a targeted structured restatement beside the decision restores its influence. This is the actionable contrast, and it points at where the information sits relative to the decision, not at how much context is provided.

Framework Shift

Before (retrieval as the proxy):
  long-context AI analyst
  evaluate: did it retrieve the disclosure?
  -> passes even when the disclosure has no influence
  -> "more context" and "better retrieval" assumed to help

After (retrieval and influence measured separately):
  retrieval accuracy   -> stays high
  judgment influence   -> measured, and collapses with context
  -> the gap is real, replicated, and not closed by scale
  -> fix the ARCHITECTURE: restate near the decision
     (chunk-and-summarize makes it worse)

From certifying a system by what it can produce on request, to measuring whether that information actually moves the decision, the core shift is that retrieval is an input to the pipeline and influence is its output, and only the latter is the product.

Expert Assessment

Problem choice: Excellent, and important well beyond finance. The retrieval-versus-use distinction applies to every long-context application — legal review, medical records, codebase assistants — and the field has been evaluating the accessible half because it is the measurable half.

Method maturity: The design is disciplined. Holding focal information fixed while varying only unrelated context is exactly the control needed to attribute the effect, and measuring retrieval and influence separately is what exposes a gap that either measurement alone would have hidden. The causal memory interventions are the strongest element, because identifying that the channel is joint — summaries and lookup together — is a mechanism claim rather than a correlation, and it explains why the obvious single-component fixes would not work.

Experimental integrity: Replicating in experiments that remove real disclosures from actual 10-K filings is the detail that makes this credible rather than contrived; synthetic probes would have left open the objection that the effect only appears under artificial conditions. Reporting that more capable models merely postpone the collapse is a result that cuts against the field’s default optimism and is welcome for it. The scope is financial judgment, which is well suited to the question but leaves open how the gap behaves in tasks with shorter decision horizons.

Writing quality: The separation of “what it can retrieve” from “whether retrieved information affects judgments” is stated in the abstract and drives the whole paper, which is exactly right. The result would be even more useful with a practical diagnostic: a procedure an engineer could run on their own pipeline to measure influence rather than retrieval, since the paper’s contribution is precisely that such a test is what everyone is missing.

Verdict: strong accept — it names a gap that is invisible to current evaluation practice, demonstrates it with real filings, explains the transmission mechanism, and points at an architectural fix rather than a scaling one.

Takeaways

• Measure influence, not retrieval. For any long-context system, test whether the information you care about actually changes the output — retrieval accuracy will not detect its absence.
• Do not expect scale to close this. More capable models postpone the collapse to longer contexts; they do not remove it, so capability is not a substitute for evaluation.
• Be suspicious of chunk-and-summarize for decision-critical content. It evicts the material that matters, and the loss is invisible because the summary is fluent.
• Put critical information adjacent to the decision. A structured restatement next to the point of judgment restored influence where more context and better retrieval did not.