Experience-RAG Skill: Teaching Agents to Choose the Right Retrieval Strategy

Paper: 2605.03989 Authors: Dutao Zhang, Tian Liao Categories: cs.AI

The Gap

RAG systems typically commit to one retrieval pipeline at design time. You pick BM25, or dense retrieval, or hybrid, and that’s what every query gets. But factoid questions want keyword matching, multi-hop reasoning needs iterative retrieval, and scientific claims demand citation-aware search. Adaptive-RAG introduced query-time routing, but it’s still a monolithic component baked into the workflow. The gap: retrieval strategy selection is treated as infrastructure plumbing rather than a modular capability an agent can learn and reuse.

Problem: One retrieval pipeline for all tasks
   |
   v
Assumption: Different tasks need different retrieval strategies
   |
   v
Method: Pluggable skill layer that selects strategies via experience memory
   |
   v
Evidence: nDCG@10 = 0.8924 across factoid/multi-hop/scientific tasks
   |
   v
Conclusion: Strategy selection can be encapsulated as reusable agent skill

The Increment

One sentence: Before this paper, retrieval strategy was infrastructure you configured; after, it’s a skill an agent can learn and swap in.

Core Mechanism

Experience-RAG Skill sits between the agent and a pool of retrievers. When a query arrives, the skill analyzes the “scene” (task type, query structure, domain), consults an experience memory of past strategy-performance pairs, selects a retrieval strategy, executes it against the retriever pool, and returns structured evidence to the agent. The experience memory is a lookup table: given scene features, which strategy worked before?

The skill doesn’t invent new retrievers. It orchestrates existing ones. The retriever pool might contain BM25, dense retrieval, hybrid search, or iterative multi-hop methods. The skill’s job is pattern matching: “This looks like a multi-hop question, last time we used iterative retrieval and it worked, let’s do that again.”

Agent query
    |
    v
+-------------------+
| Experience-RAG    |
| Skill             |
|                   |
| 1. Scene analysis |<--- (task type, query structure)
| 2. Memory lookup  |<--- (experience: scene -> strategy)
| 3. Strategy pick  |
+-------------------+
    |
    v
Retriever pool (BM25 | Dense | Hybrid | Iterative)
    |
    v
Structured evidence -> Agent

Think of it like a restaurant kitchen. The agent is the head chef who decides what dish to make. The retrievers are specialized stations: grill, sauté, pastry. Experience-RAG Skill is the sous chef who’s worked every station and knows which one to send each order to. A steak? Grill. A soufflé? Pastry. The sous chef doesn’t cook—they route based on pattern recognition from past service. The head chef doesn’t need to know which station handles what; they just ask the sous chef and get back prepped ingredients. The skill encapsulates routing logic so the agent can focus on reasoning, not retrieval plumbing.

Key Concepts

• Scene: The task context that determines retrieval needs. A scene isn’t just the query text—it’s the inferred task type (factoid vs multi-hop vs verification), domain (news vs science), and structural cues (question length, entity count). Scene analysis is lightweight classification: “Does this query have multiple entities and a causal connector? Probably multi-hop.” The skill doesn’t need perfect classification; it needs enough signal to distinguish retrieval preferences.

• Experience Memory: A mapping from scenes to strategies, built from past performance. Not a neural network—just a table. “For multi-hop science questions, iterative retrieval got nDCG 0.91 last time.” When a new query arrives, the skill finds the closest past scene and reuses its strategy. The memory grows over time as the agent encounters new task types. It’s collaborative filtering for retrieval: what worked for similar queries?

• Pluggable Skill: A modular capability the agent can invoke without knowing its internals. The agent calls retrieve(query) and gets back evidence. Whether that evidence came from BM25 or iterative search is hidden. The skill interface is stable even as the retriever pool changes. You can add a new retriever, update the experience memory, and the agent’s code doesn’t change. This is the key architectural move: retrieval strategy becomes a black-box service, not tangled logic in the agent’s control flow.

Framework Shift

Before (mainstream approach):        After (this paper):

Agent                                Agent
  |                                    |
  | (hard-coded strategy)              | retrieve(query)
  v                                    v
Single retriever pipeline            Experience-RAG Skill
  |                                    |
  v                                    | (scene analysis + memory lookup)
Evidence                               v
                                     Retriever pool
                                       |
                                       v
                                     Evidence

From monolithic retrieval pipeline to modular skill layer, the core shift is treating strategy selection as a learned, reusable capability rather than design-time configuration.

Expert Assessment

Problem choice: Real gap. RAG systems do struggle with heterogeneous tasks, and hard-coding strategy selection into agent logic is brittle. The framing as a “skill” is clever—it aligns with the agent-oriented architecture trend and makes the contribution feel like a design pattern, not just another routing method.

Method maturity: Mostly engineering, not algorithmic novelty. Scene analysis is heuristic classification, experience memory is a lookup table, and the retriever pool is off-the-shelf. The insight is architectural: encapsulation matters. But the paper doesn’t explore how the experience memory is built (manual annotation? online learning?) or how it generalizes to unseen scenes. The “experience” framing suggests learning, but the implementation seems closer to a hand-tuned dispatch table.

Experimental integrity: Baselines are fair (single retrievers, Adaptive-RAG), and nDCG@10 of 0.8924 is solid. But the evaluation is limited to three BeIR tasks. No ablation on scene analysis quality, no failure case analysis, no comparison of memory update strategies. The claim that this “remains competitive with Adaptive-RAG” undersells it—if they’re tied, why is the skill framing better? The paper needs to show where encapsulation pays off beyond raw performance.

Writing quality: The abstract and intro are crisp, but the method section is vague. How exactly is scene analysis implemented? What features go into the memory lookup? The paper reads like a position statement with preliminary results rather than a complete system description. Section 3 (Method) should be twice as long with concrete algorithms and data structures.

Verdict: weak accept — The architectural insight (retrieval strategy as pluggable skill) is valuable and the results are promising, but the paper needs deeper technical exposition and broader evaluation to be convincing.

Takeaways

The core idea transfers: encapsulate strategy selection as a reusable module with a stable interface. If you’re building any system where different subtasks need different tools (code generation with multiple LLMs, data pipelines with multiple transforms, UI rendering with multiple frameworks), don’t hard-code the dispatch logic into the orchestrator. Build a “skill” layer that maps task characteristics to tool choices, and let that layer evolve independently. The experience memory pattern is also portable: maintain a lightweight lookup table of (context, tool, performance) triples and use nearest-neighbor matching for new contexts. It’s not fancy, but it decouples tool selection from business logic, which is often the right move.