Static Relevance Is a Lie: Bridge Documents in Agentic Search

Paper: 2607.15253 Authors: Debayan Mukhopadhyay, Utshab Kumar Ghosh, Shubham Chatterjee Categories: cs.IR, cs.CL

The Gap

Retrieval-augmented generation has a dirty secret: every retrieval system is trained and evaluated on the assumption that a document’s usefulness can be scored in isolation. You hand a reader model a question and a single document, and you measure whether the answer got better. This is how we train rerankers, how we evaluate search pipelines, and how we build RAG benchmarks. The assumption holds when a model reads one document for one question.

It breaks when the language model works as an agent—issuing multiple queries, reading documents across turns, and building a chain of reasoning. In that setting, a document’s real value often lies not in what it says about the current question, but in what it enables the agent to do next: a new entity, a redirect, a pivot in the search trajectory. No static relevance score captures this. Prior work on multi-hop retrieval has focused on whether the right documents appear in the top-k, not on whether they are causally necessary for the agent’s trajectory. This paper measures that gap directly, and finds it is not a gap—it is a chasm.

Static RAG assumption
  |
  v
Doc usefulness = f(doc, question)
  |
  v
Works for single-turn reading
  |
  v
Breaks in agentic multi-turn search
  |
  v
Agent reads doc --> gets entity --> redirects query --> finds answer
  |                                                       ^
  +--- Doc had no direct answer, but was causally essential
  |
  v
Delete doc --> re-run trajectory --> answer degrades
  |
  v
Static score and causal utility are independent (rho = -0.03)

The Increment

One sentence: Before this paper, we scored retrieval documents as if each one existed in a vacuum; after this paper, we have empirical evidence and a measurement framework showing that in agentic search, a third of all useful documents are invisible to static evaluation and earn their keep by redirecting the agent’s next move.

Core Mechanism

The paper introduces a counterfactual deletion methodology. Take a ReAct-style agent running over HotpotQA multi-hop questions. Let it complete its full trajectory—multiple queries, multiple documents read, final answer produced. Then, for every single document the agent read during that trajectory, delete it and re-run the entire trajectory from the point of deletion onward, keeping all prior context intact. Compare the original run against the counterfactual run on three axes: did the final answer quality change? Did the quality of the agent’s next retrieved set change? Did the number of turns needed change? The combination of these three deltas produces a Counterfactual Trajectory Utility (CTU) score for that document.

Cross CTU against Static RAG Utility (SRU)—the conventional score you would assign by giving the reader model that document and question in isolation—and you get the paper’s central result. Over 23,322 document observations, the Spearman rank correlation between SRU and CTU is -0.026. They are close to statistically independent. Knowing that a document looks useful to a static reader tells you essentially nothing about whether removing it would derail the agent’s trajectory.

The second experiment pinpoints the mechanism using Observable Entity Relevance (OER). Entities that discriminate relevant from non-relevant candidates—entities that appear in the correct answer or the golden reasoning chain—show up in the agent’s next query 4.02 times more often when they come from a causally useful document versus a non-useful one. The bridge document’s job is not to answer the question; it is to hand the agent a discriminative entity that redirects the search toward the answer.

Full agent trajectory (HotpotQA, ReAct agent)
  |
  |  For each document d_i in trajectory:
  |
  v
Delete d_i --> re-run trajectory from turn i
  |
  v
Measure three deltas vs original run:
  |         |         |
  v         v         v
Answer    Retrieval   Turn
quality   quality     count
  |         |         |
  +----+----+----+----+
       |
       v
CTU score for d_i
  |
  v
Cross with SRU (static score) --> rho = -0.03
  |
  v
Bridge documents: high CTU, low SRU (~1/3 of all docs)
  |
  v
Mechanism: discriminative entities appear in next query
  |          4.02x more often from bridge docs
  v
OER analysis (n = 227,139 entities)

Core Mechanism (continued)

Here is a structural metaphor. Imagine you are solving a murder mystery. You interview witnesses one by one. A static relevance score would judge each witness by whether they directly tell you who the killer is. But in reality, Witness #3 might only tell you that the suspect was seen at the train station—and that detail is what makes you think to interview the station manager next, who is the one who actually identifies the killer. Witness #3 is a bridge witness. Under static evaluation, Witness #3 looks useless: they never named the killer. Under counterfactual evaluation, Witness #3 is indispensable: remove them and you never find the station manager, never solve the case.

Every component of the method maps to the mystery:

• The agent trajectory is your full investigation path across all interviews.
• Static RAG Utility is your judgment of a witness based solely on whether they named the killer.
• Counterfactual Trajectory Utility is your judgment after replaying the investigation without that witness and checking if the case gets solved.
• Bridge documents are the witnesses who seem uninformative but redirect your investigation.
• Discriminative entities are the specific details (train station) that cause you to pivot your line of inquiry.

The paper’s contribution is measuring, at scale, how often the static judgment and the counterfactual judgment disagree—and the answer is: almost always, for about a third of the documents.

Key Concepts

• Counterfactual Trajectory Utility (CTU): Think of it like this. You bake a cake using ten ingredients. To figure out which ingredient matters, you bake the cake ten more times, each time leaving one ingredient out. If leaving out the salt makes the cake taste terrible, salt has high CTU even though nobody would call salt “the star of the recipe.” CTU measures a document’s importance by what happens when you remove it from a multi-step process, not by what it contributes in isolation.

• Bridge Document: A document that scores low on static relevance—you hand it to a reader model with a question and the model says “meh”—but removing it from the agent’s trajectory causes the agent to fail or take a much longer path. It is a stepping stone, not a destination. It contains a term, an entity, or a phrase that the agent picks up and uses to reformulate its next query. Without it, the agent never pivots in the right direction. Roughly one in three documents the agent reads are bridge documents.

• Discriminative Entity: An entity (a person name, a place, a date) that appears in a relevant document but not in non-relevant ones. If the question is about “which country” and a document mentions “Treaty of Utrecht,” that entity is discriminative—it narrows the search space. The paper shows that bridge documents supply these discriminative entities, and the agent latches onto them to redirect its next query. This is the causal mechanism: not “the document answers the question” but “the document gives the agent a clue for its next move.”

Framework Shift

Before (mainstream approach):          After (this paper):

  Question + Doc                        Agent trajectory (many turns)
       |                                     |
       v                                     v
  Reader model                         Counterfactual deletion
       |                                for each doc in trajectory
       v                                     |
  Score: useful or not?                      v
       |                                Three deltas measured:
       v                                answer + retrieval + turns
  Train/eval retrieval                       |
  systems on this score                      v
                                         CTU score
                                         crossed with SRU
                                              |
                                              v
                                         Bridge docs identified
                                         Mechanism: entity redirection

From evaluating documents in isolation to evaluating them by their causal role in a trajectory, the core shift is treating retrieval utility as a counterfactual property of a process rather than a static property of a document-question pair.

Expert Assessment

Problem choice: This is a real and underexplored gap. The field has been optimizing retrieval for single-turn RAG while agents are rapidly becoming the dominant deployment pattern. The disconnect between static evaluation and agentic utility is not something the authors invented—it has been hiding in plain sight. Good timing, important question.

Method maturity: Counterfactual deletion is brute force but intellectually clean. The cost is enormous: you need to re-run the full trajectory for every document deletion, which multiplies compute by the number of documents per trajectory. The authors acknowledge this by limiting to 1,000 questions, but it does limit scalability claims. A lighter-weight proxy for CTU would be a significant contribution. The OER analysis is more elegant and points toward scalable alternatives.

Experimental integrity: HotpotQA is a reasonable choice for multi-hop questions, but it is still a constructed benchmark. The agent is ReAct-style, which is one architecture among many. The rho = -0.026 result is striking, but the paper would benefit from confidence intervals and robustness checks across different agent architectures and different retrieval models. The bridge cell percentage of 27.2% is compelling but depends on the axis discretization. The OER analysis with n = 227,139 is well-powered. No red flags, but generalization is an open question.

Writing quality: The abstract is excellent—clear, precise, no hand-waving. The related work section is thin; the paper would benefit from deeper engagement with the causal inference literature and with prior work on search session analysis. Section 4 (the OER analysis) feels somewhat grafted on; integrating it more tightly with the counterfactual narrative would strengthen the whole argument. The core insight is stated once and demonstrated well, but the paper does not fully explore the implications for retrieval system design.

Verdict: weak accept — The measurement is clean and the finding is important, but the single-benchmark, single-agent-architecture scope limits how far the conclusions can be generalized. The insight that static relevance is nearly independent of causal utility in agentic search is worth the community’s attention.

Takeaways

1. If you are building retrieval for agents, stop relying on static relevance scores to decide what to retrieve. You need to evaluate documents by their downstream trajectory effect, not their isolated answer contribution.

2. Counterfactual deletion is the gold standard metric for agentic retrieval, but it is expensive. The OER measure (tracking discriminative entity overlap between documents and the agent’s next query) is a cheap, scalable proxy that correlates with causal utility. Consider instrumenting your agent to log entity-query overlaps as a lightweight utility signal.

3. The “bridge document” framing is transferable. In any multi-step reasoning system—not just retrieval—there are intermediate artifacts that look useless in isolation but are causally essential for the process. If you evaluate pipeline components in isolation, you will systematically undervalue the connectors.

4. When training retrieval models for agentic use, consider trajectory-level rewards rather than document-level relevance. The paper’s evidence suggests that optimizing for static relevance may actually be anti-correlated with the behavior you want.