EEVEE: Real-World Test-Time Prompt Learning for Self-Improving LLM Agents

Paper: 2606.11182 Authors: Weixian Xu, Shilong Liu, Mengdi Wang Categories: cs.LG, cs.AI

The Gap

Existing test-time prompt learning (TTPL) methods — GEPA, ACE, and earlier fine-tuning based approaches — were designed and evaluated in single-dataset settings. They assume the agent sees a homogeneous stream of questions from one benchmark. But in the wild, an LLM agent encounters a chaotic mix of tasks: math word problems, code generation, factual QA, creative writing — all interleaved unpredictably.

When you naively apply a single learned prompt to such a heterogeneous stream, performance degrades because the prompt overfits to one distribution while hurting others — cross-dataset interference. The core missing piece is a mechanism that can dynamically adapt prompts to the current input without any labels, while maintaining efficiency.

Problem: LLM agent must handle real-world multi-dataset streams
           |
           v
Assumption: One prompt fits all (current TTPL assumption)
           |
           v
Cross-dataset interference + performance drop
           |
           v
Method: Router partitions inputs into task clusters
        Router + prompts co-evolve (interleaved learning)
           |
           v
Evidence: +10.38 to +24.32 avg multi-benchmark over base LLMs
          Outperforms SOTA by 37-48%
           |
           v
Conclusion: Multi-dataset TTPL is viable and practical

The Increment

One sentence: Before EEVEE, test-time prompt learning worked only when each deployment had a single dataset; after EEVEE, prompts can adapt to a real-world stream of unknown, mixed tasks without ever seeing labels.

Core Mechanism

EEVEE has three moving parts: a Router, a Prompt Cluster, and a Co-evolution Loop.

Data flows in one pass at inference time. An input query arrives. The Router (a lightweight neural network) encodes the input and produces a soft assignment over a fixed number of “task clusters”. Each cluster has its own learned prompt token(s). The router then selects the top-K clusters, weights their prompts, and the LLM generates the answer using the weighted combination.

The critical innovation is how the Router and Prompts are learned — they are interdependent. The Router needs good prompts to know which cluster a query belongs to, but the prompts need good routing to avoid cross-task interference. EEVEE solves this with an interleaved training loop: freeze the router, update prompts; then freeze prompts, update router. This co-evolution converges to a partition of the input space where each prompt specialises.

Input query -> [Encoder] -> Router (softmax over K clusters)
                    |
                    v
        cluster #1  cluster #2  ...  cluster #K
             |          |
           prompt#1   prompt#2
             \          /
              v        v
         Weighted prompt -> LLM -> Output

Training phases:
Phase A: freeze Router, optimize prompts (each cluster's prompt sees only assigned queries)
Phase B: freeze prompts, optimize Router (maximizes likelihood of correct cluster for each query)
Repeat until convergence

Structural Metaphor: A Library with Drifting Sections

Imagine a university library where new books arrive every hour, but the library’s sections (Mathematics, History, etc.) keep slowly moving — because the head librarian is constantly reassigning shelves based on newer books. The librarian (Router) categorises each new book and directs it to the appropriate section. Each section (Cluster) has a designated “lending policy” (Prompt) that tells patrons how to best find information in that section. But here’s the catch: the sections and policies are learned together. If the librarian sends a physics book to the math section, the math policy won’t work well. So the librarian must “practice” by seeing how patrons respond, and the sections must adjust their policies based on the books they receive. EEVEE’s co-evolution is this librarian-policy dance, interleaving updates until the library maps the real stream of topics perfectly.

Key Concepts

• Test-time Prompt Learning (TTPL): The ability to update the prompt at inference time without any ground-truth labels. Typically done by using the LLM’s own predictive confidence or consistency as a pseudo-label. TTPL is crucial when you cannot collect labelled data after deployment. EEVEE extends TTPL from single-stream to multi-stream.

• Router-Prompt Co-evolution: The chicken-and-egg problem that arises when you need both a classifier (router) and specialised prompt (policy) but neither is known in advance. The solution: alternate between optimising one while fixing the other, akin to Expectation-Maximisation. This ensures each prompt “owns” a coherent cluster of tasks.

• Cross-dataset Interference: When you train a single prompt on multiple task distributions, the prompt becomes a compromise that fails on all. EEVEE avoids this by having separate prompts for separate clusters, thus preserving specialisation.

Framework Shift

Before (mainstream TTPL): single data stream -> single learned prompt -> answer.

Stream (all Math) -> [LLM with prompt P] -> answer

After (EEVEE): multi-dataset stream -> Router -> multiple cluster prompts -> weighted answer.

Stream (Math, Code, QA, ...) -> [Router] -> cluster[1] : prompt[1]
                                               cluster[2] : prompt[2]  } weighted combination -> [LLM] -> answer
                                               cluster[3] : prompt[3]

One sentence: From a one-size-fits-all prompt to a dynamically partitioned prompt bank, the core shift is introducing structural specialisation through a learnable router that adapts to the input stream at test time.

Expert Assessment

Problem choice: Real and practical. The single-dataset assumption of prior TTPL work is limiting; multi-dataset streams are the default in production. This paper fills a genuine gap.

Method maturity: Clever insight — routing and co-evolution are not new individually, but their combination for TTPL is novel. The authors don’t claim to invent router or EM-style learning, they apply them to a new problem with careful engineering. Could there be simpler solutions? Maybe online clustering + handcrafted task prompts? But that would require human effort; end-to-end tuning is more scalable.

Experimental integrity: Baselines are appropriate (GEPA, ACE, single-prompt baselines), and results are strong. I’d like to see more analysis on the router’s learned clusters — are they semantically meaningful? The paper focuses on average scores, which is fine but hides per-task variability. No obvious red flags.

Writing quality: The paper is clear on motivation and results but rushes the router architecture details. A dedicated section on what the router sees (input embedding? raw text?) and how prompts are stored (continuous vectors? strings?) would strengthen reproducibility.

Verdict: weak accept — a solid contribution that solves a real problem with a neat method, but the novelty lies in the combination rather than a breakthrough in any single component.

Takeaways

1. Routing as a building block for adapter modularity: The idea of using a router to partition the input space and assign specialised components is transferable to any multi-task learning scenario (e.g., multi-domain fine-tuning, multi-skill LLM agents). The co-evolution trick ensures the components actually specialise.

2. Interleaved training for interdependent modules: Whenever two modules have a chicken-and-egg dependency (router and prompts, or root and leaves in a tree), alternate updates can converge effectively. This is a reusable design pattern.

3. No labels needed: The entire EEVEE pipeline operates with no human annotation after initial deployment, making it suitable for continuous self-improvement in production.