Paper: 2605.15184 Authors: Sahil Sen, Akhil Kasturi, Elias Lumer, Anmol Gulati, Vamse Kumar Subbiah Categories: cs.CL
The Gap
Everyone’s building RAG agents now. Vector retrieval is the default choice—embed your corpus, embed the query, find nearest neighbors. The assumption: semantic similarity beats keyword matching. But here’s what nobody’s systematically tested: when you wrap retrieval in an agent loop with tool calls, does vector search still win? And does it matter whether the agent sees tool results inline versus reading them from files?
Prior work evaluates retrieval in isolation or in simple RAG pipelines. This paper asks: what happens when retrieval becomes one tool among many in an agentic workflow, where the model orchestrates multiple calls and reasons over accumulated context?
Problem: RAG agents use vector retrieval by default
|
v
Assumption: Vector > grep in agent loops (untested)
|
v
Method: Compare grep vs vector across 4 agent harnesses
+ inline vs file-based tool results
+ progressive noise injection
|
v
Evidence: Grep wins in most configurations
Harness architecture dominates retrieval choice
|
v
Conclusion: Agent infrastructure matters more than
retrieval algorithm alone
The Increment
One sentence: Before this paper, we assumed vector retrieval was superior for agentic search; after, we know that grep often wins and that harness design (how tools present results) matters more than retrieval strategy.
Core Mechanism
The paper runs two experiments. Experiment 1 compares grep and vector retrieval on 116 questions from LongMemEval using four agent harnesses: a custom harness called Chronos, plus Claude Code, Codex, and Gemini CLI. Each harness is tested with two tool-result presentation modes: inline (results appear directly in the conversation) and file-based (results are written to files that the model must read separately).
Experiment 2 isolates the effect of noise. Starting with the same questions, the authors progressively inject unrelated conversation history around each query. This simulates real-world scenarios where agents accumulate context from prior interactions. They test grep-only and vector-only retrieval as noise increases from 0% to 75% of the context window.
Experiment 1: Retrieval x Harness x Presentation
Query --> [Harness] --> Tool Call --> [Retrieval]
| |
| v
| grep / vector
| |
v v
Inline Result File-based Result
| |
+----------> Model <---------+
|
v
Answer
Experiment 2: Retrieval x Noise Level
Query + Noise (0% -> 75%) --> [Harness] --> Tool Call
|
v
grep / vector
|
v
Answer
Think of this like testing a chef’s knife versus an electric slicer in different kitchens. The knife is grep—simple, direct, no preprocessing. The slicer is vector retrieval—requires setup (embedding), promises precision. But here’s the twist: you’re not just comparing tools in isolation. You’re testing them in four different kitchens (harnesses), each with different counter layouts (inline vs file-based results). Then you add distractions—other cooks moving around, background noise (irrelevant context). The question isn’t “which tool is sharper?” but “which tool works better in this specific kitchen setup, under these specific conditions?”
The grep knife often wins because it’s predictable: you know exactly what you’ll get when you search for a string. The vector slicer can miss if the embedding space doesn’t align well with the query, especially when the model has to navigate through files or filter noise.
Key Concepts
-
Agent Harness: The infrastructure that wraps an LLM and manages its interaction with tools. Think of it as the operating system for an agent. It decides how tool calls are formatted, how results are returned (inline text vs file paths), how context is managed, and how the conversation flows. Different harnesses (Claude Code, Codex, Gemini CLI, Chronos) implement these decisions differently, leading to performance variations even when using the same underlying model and retrieval method. It’s not just a thin wrapper—it’s the entire execution environment that shapes what the model sees and how it reasons.
-
Inline vs File-based Tool Results: When a tool returns data, there are two ways to present it to the model. Inline means the result appears directly in the conversation as text—the model sees it immediately without additional action. File-based means the tool writes results to a file and returns a path—the model must explicitly read that file in a subsequent step. This seems like a minor implementation detail, but it fundamentally changes the agent’s cognitive load. Inline is like someone handing you a document already open to the relevant page. File-based is like someone giving you a filing cabinet location—you have to go fetch it yourself. The extra step introduces failure modes: the model might forget to read the file, read the wrong file, or misinterpret the path.
-
Grep in Agent Context: Grep here isn’t just the Unix command—it’s a retrieval strategy that searches for exact or fuzzy string matches without semantic understanding. In an agent loop, grep becomes a tool the model can call with a query string. The advantage: deterministic, transparent, no embedding drift. The model knows that if it searches for “user authentication,” it will get passages containing those exact words. Vector retrieval, by contrast, might return passages about “login security” or “credential management”—semantically related but not lexically matching. In noisy contexts or when the model needs to verify specific facts, grep’s literalness becomes a feature, not a bug.
Framework Shift
Before (mainstream approach): After (this paper):
Query Query
| |
v v
Embed Query Choose: Embed or Grep?
| |
v +---> Grep (often wins)
Vector Search |
| +---> Vector (sometimes wins)
v |
Top-K Results v
| Results via Harness
v |
Model Generates Answer +---> Inline (simpler)
|
+---> File-based (harder)
|
v
Model Reasons & Answers
|
v
Harness Architecture
Dominates Performance
From “vector retrieval is the default” to “grep often wins, and how you present results matters more than which retrieval you choose.”
Expert Assessment
Problem choice: Real gap. The RAG literature has been obsessed with embedding models and vector databases, but nobody’s systematically tested whether that sophistication pays off in agentic workflows where retrieval is one tool among many. The focus on harness architecture is especially valuable—it’s the kind of unglamorous infrastructure question that practitioners face but researchers ignore.
Method maturity: Solid empirical work, not trying to be clever. The experimental design is straightforward: compare A vs B across multiple dimensions, control for confounds, progressively stress-test with noise. The choice to test multiple harnesses (including provider CLIs, not just a custom setup) strengthens generalizability. However, 116 questions is a small sample, and LongMemEval is a specific domain (long-context QA). Would be stronger with more diverse tasks.
Experimental integrity: Baselines are fair—grep and vector retrieval are both reasonable choices, and the harnesses are real systems people use. The noise injection in Experiment 2 is a clever way to simulate real-world context accumulation. One concern: the paper doesn’t deeply analyze *why grep wins. Is it because vector embeddings are misaligned with the task? Because file-based presentation breaks the agent’s reasoning? The results are clear, but the causal story is underdeveloped.
Writing quality: The abstract and introduction are clear, but the results section could be tighter. There’s a lot of “grep scored X, vector scored Y” without enough interpretation. The discussion of harness differences is buried—it should be front and center, since that’s the most surprising finding. The paper would benefit from a “failure analysis” section showing specific examples where grep succeeded and vector failed, or vice versa.
Verdict: weak accept — Addresses a real gap with solid empirical evidence, but the analysis of *why these results occur is shallow, and the sample size limits generalizability.
Takeaways
If you’re building an agent system, don’t default to vector retrieval without testing grep first. Especially if your agent needs to verify specific facts or handle noisy contexts, grep’s determinism can outweigh vector search’s semantic flexibility.
More importantly: invest in your harness architecture. How you present tool results to the model (inline vs file-based) has a bigger impact on performance than which retrieval algorithm you choose. If you’re seeing inconsistent agent behavior, look at the infrastructure layer before blaming the model or the retrieval method.
For researchers: this paper shows that agent evaluation needs to account for the full stack, not just the model or the retrieval component in isolation. A method that works in a clean RAG pipeline might fail in an agentic loop with file-based tools and accumulated context.
论文: 2605.15184 作者: Sahil Sen, Akhil Kasturi, Elias Lumer, Anmol Gulati, Vamse Kumar Subbiah 分类: cs.CL
缺口
现在大家都在搭 RAG 智能体。
向量检索是默认选择——把语料库嵌入，把查询嵌入，找最近邻。
假设是：语义相似度胜过关键词匹配。
但没人系统地测试过：当你把检索包在一个带工具调用的智能体循环里，向量搜索还能赢吗？
智能体是内联看到工具结果，还是从文件里读，这有区别吗？
此前的工作要么孤立地评估检索，要么在简单的 RAG 管道里评估。
这篇论文问的是：当检索成为智能体工作流中众多工具之一，模型要编排多次调用、对累积的上下文推理时，会发生什么？
问题：RAG 智能体默认用向量检索
|
v
假设：在智能体循环中向量 > grep（未经测试）
|
v
方法：在 4 个智能体框架中比较 grep 和向量
+ 内联 vs 基于文件的工具结果
+ 渐进式噪声注入
|
v
证据：grep 在大多数配置中获胜
框架架构主导检索选择
|
v
结论：智能体基础设施比检索算法本身更重要
增量
一句话: 这篇论文之前，我们假设向量检索在智能体搜索中更优；
之后，我们知道 grep 往往获胜，且框架设计（工具如何呈现结果）比检索策略更重要。
核心机制
论文做了两个实验。
实验 1 在 LongMemEval 的 116 个问题上比较 grep 和向量检索，使用四个智能体框架：一个叫 Chronos 的自定义框架，加上 Claude Code、Codex 和 Gemini CLI。
每个框架测试两种工具结果呈现模式：内联（结果直接出现在对话中）和基于文件（结果写入文件，模型必须单独读取）。
实验 2 隔离噪声的影响。
从相同的问题开始，作者在每个查询周围渐进式注入无关的对话历史。
这模拟了智能体从先前交互中累积上下文的真实场景。
他们测试纯 grep 和纯向量检索，噪声从上下文窗口的 0% 增加到 75%。
实验 1：检索 x 框架 x 呈现方式
查询 --> [框架] --> 工具调用 --> [检索]
| |
| v
| grep / 向量
| |
v v
内联结果 基于文件的结果
| |
+-------> 模型 <-------+
|
v
答案
实验 2：检索 x 噪声水平
查询 + 噪声 (0% -> 75%) --> [框架] --> 工具调用
|
v
grep / 向量
|
v
答案
把这想象成在不同厨房里测试厨师刀和电动切片机。
刀是 grep——简单、直接、无需预处理。
切片机是向量检索——需要设置（嵌入），承诺精确。
但转折来了：你不是孤立地比较工具。
你在四个不同的厨房（框架）里测试它们，每个厨房的台面布局不同（内联 vs 基于文件的结果）。
然后你添加干扰——其他厨师走来走去，背景噪音（无关上下文）。
问题不是”哪个工具更锋利？”
而是”在这个特定的厨房设置下，在这些特定条件下，哪个工具更好用？”
grep 刀往往获胜，因为它可预测：你搜索一个字符串时，你确切知道会得到什么。
向量切片机可能会错过，如果嵌入空间与查询对齐不好，尤其是当模型必须浏览文件或过滤噪声时。
关键概念
- 智能体框架（Agent Harness）: 包裹 LLM 并管理其与工具交互的基础设施。
把它想象成智能体的操作系统。
它决定工具调用如何格式化、结果如何返回（内联文本 vs 文件路径）、上下文如何管理、对话如何流动。
不同的框架（Claude Code、Codex、Gemini CLI、Chronos）对这些决策的实现不同，导致性能差异，即使使用相同的底层模型和检索方法。
它不只是一个薄包装器——它是塑造模型所见和推理方式的整个执行环境。
- 内联 vs 基于文件的工具结果: 当工具返回数据时，有两种方式呈现给模型。
内联意味着结果直接作为文本出现在对话中——模型无需额外操作就能立即看到。
基于文件意味着工具将结果写入文件并返回路径——模型必须在后续步骤中显式读取该文件。
这看起来像个小实现细节，但它从根本上改变了智能体的认知负荷。
内联就像有人把文档已经翻到相关页递给你。
基于文件就像有人给你一个档案柜位置——你得自己去取。
额外的步骤引入了失败模式：模型可能忘记读文件、读错文件或误解路径。
- 智能体上下文中的 Grep: 这里的 grep 不只是 Unix 命令——它是一种检索策略，搜索精确或模糊的字符串匹配，没有语义理解。
在智能体循环中，grep 成为模型可以用查询字符串调用的工具。
优势：确定性、透明、无嵌入漂移。
模型知道如果它搜索”用户认证”，它会得到包含这些确切词的段落。
相比之下，向量检索可能返回关于”登录安全”或”凭证管理”的段落——语义相关但词汇不匹配。
在嘈杂的上下文中，或当模型需要验证特定事实时，grep 的字面性成为特性，而非缺陷。
框架转变
之前（主流方法）： 之后（本文方法）：
查询 查询
| |
v v
嵌入查询 选择：嵌入还是 Grep？
| |
v +---> Grep（往往获胜）
向量搜索 |
| +---> 向量（有时获胜）
v |
Top-K 结果 v
| 通过框架返回结果
v |
模型生成答案 +---> 内联（更简单）
|
+---> 基于文件（更难）
|
v
模型推理并答案
|
v
框架架构主导性能
从”向量检索是默认选择”到”grep 往往获胜，且如何呈现结果比选择哪种检索更重要”。
专家评审
选题眼光: 真实的缺口。
RAG 文献一直痴迷于嵌入模型和向量数据库，但没人系统地测试过这种复杂性在智能体工作流中是否值得，在那里检索只是众多工具之一。
对框架架构的关注尤其有价值——这是实践者面临但研究者忽视的那种不起眼的基础设施问题。
方法成熟度: 扎实的实证工作，不试图耍聪明。
实验设计直截了当：在多个维度上比较 A vs B，控制混淆因素，用噪声渐进式压力测试。
选择测试多个框架（包括提供商 CLI，不只是自定义设置）增强了普适性。
但是，116 个问题是小样本，LongMemEval 是特定领域（长上下文问答）。
如果有更多样化的任务会更强。
实验诚意: 基线公平——grep 和向量检索都是合理选择，框架是人们使用的真实系统。
实验 2 中的噪声注入是模拟真实世界上下文累积的巧妙方法。
一个担忧：论文没有深入分析 为什么 grep 获胜。
是因为向量嵌入与任务不对齐？
因为基于文件的呈现打断了智能体的推理？
结果很清楚，但因果故事欠发达。
写作功力: 摘要和引言清晰，但结果部分可以更紧凑。
有很多”grep 得分 X，向量得分 Y”，没有足够的解释。
对框架差异的讨论被埋没了——它应该放在最前面，因为那是最令人惊讶的发现。
论文会受益于一个”失败分析”部分，展示 grep 成功而向量失败的具体例子，反之亦然。
判决: 弱接收 — 用扎实的实证证据解决了真实的缺口，但对 *为什么 出现这些结果的分析浅薄，样本量限制了普适性。
要点总结
如果你在构建智能体系统，不要在没测试 grep 的情况下默认使用向量检索。
特别是如果你的智能体需要验证特定事实或处理嘈杂的上下文，grep 的确定性可以超过向量搜索的语义灵活性。
更重要的是：投资你的框架架构。
如何向模型呈现工具结果（内联 vs 基于文件）对性能的影响比你选择哪种检索算法更大。
如果你看到不一致的智能体行为，在责怪模型或检索方法之前，先看基础设施层。
对研究者：这篇论文表明，智能体评估需要考虑整个堆栈，而不仅仅是孤立的模型或检索组件。
在干净的 RAG 管道中有效的方法，在带有基于文件的工具和累积上下文的智能体循环中可能会失败。