Knowledge Access Beats Model Size: Memory Augmented Routing for Persistent AI Agents

Paper: 2603.23013 Authors: Xunzhuo Liu, Bowei He, Xue Liu, Andy Luo, Haichen Zhang, Huamin Chen Categories: cs.CL

Abstract

Production AI agents frequently receive highly repetitive user-specific queries — up to 47% are semantically similar to prior interactions — yet each query is processed with the same computational cost. This work proposes a memory-augmented inference framework where a lightweight 8B model leverages retrieved conversational context to answer via a low-cost inference path. Without any training or labeled data, this achieves 30.5% F1, recovering 69% of a full-context 235B model’s performance while reducing cost by 96%. The central finding: a 235B model without memory (13.7% F1) underperforms even a standalone 8B model (15.4% F1), demonstrating that for user-specific queries, access to relevant knowledge outweighs model scale.

Key Contributions

• “Knowledge access beats model size”: 235B without memory<8B without memory on user-specific queries
• 2×2 factorial study: First to study the interaction between memory and routing, revealing that memory doesn’t enable routing — it makes routing worthwhile
• 96% cost reduction: Memory-augmented 8B handles 100% of queries at small-model cost
• Zero training required: Uses only the model’s own log-probabilities as confidence signal
• Failure mode analysis: Turn-pair memories help factual recall (+28.9 F1) but hurt temporal reasoning (-3.8 F1)

The 2×2 Factorial Experiment

	No Routing	Routing (τ=0.50)
No Memory	15.4% F1	13.0% F1 (96% on 8B)
Memory	30.1% F1	30.5% F1 (100% on 8B)

What This Reveals

Without memory: Routing already keeps 96% of queries on the small model — but the small model is confidently wrong. It fabricates plausible-sounding answers about user-specific facts it has never seen.

With memory: The small model answers correctly because it has the relevant context. Memory doesn’t change the routing decision; it changes whether the routed answer is correct.

The compound effect: “Memory makes routing worthwhile” — routing provides cost savings, memory ensures those savings come with quality.

The Virtuous Cycle

In production agent workloads (47% of queries are semantically similar):

1. First query on a topic → may need expensive large model → builds memory
2. Same/similar question recurs (65% of production queries) → small model retrieves prior knowledge → answers correctly
3. Memory store grows over time → more topics covered → quality converges toward large-model quality at small-model cost

Concrete Example

User asks: “What is Caroline’s marital status?”

• Without memory (8B): Produces plausible but incorrect response. Confidence = 0.38
• With memory (8B, retrieved turn: “Q: Are you seeing anyone? A: No, I’m single”): Answers correctly. Confidence = 0.93

Failure Modes

Memory Type	Factual Recall (Single-hop)	Temporal Reasoning
Turn-pair	+28.9 F1	-3.8 F1
Hybrid (BM25 + cosine)	Additional +7.7 F1	Still limited

Turn-pair memories work well for “what” questions but poorly for “when” questions requiring temporal ordering across conversations — suggesting structured memory representations are needed for these cases.

Takeaways

• For user-specific queries, knowledge access matters more than model size — a finding with major implications for production system design
• The memory-routing interaction is non-obvious and was previously unstudied: memory doesn’t enable routing, it makes routing worthwhile
• Confident hallucination is the primary failure mode of small models without memory — they don’t know they don’t know
• The framework requires zero training, zero labeled data, and zero external classifiers — just conversational memory + log-probability confidence
• Production agent workloads are repetitive enough (47%+ similar queries) to make this approach highly effective in practice