When More Reasoning Hurts: Why Programmatic State Beats LLM Deliberation in Adversarial Environments

Paper: 2605.16205 Authors: Igor Bogdanov, Chung-Horng Lung, Thomas Kunz, Jie Gao, Adrian Taylor, Marzia Zaman Categories: cs.AI, cs.CL, cs.LG, cs.MA, eess.SY

The Gap

Compound LLM agents are being deployed in adversarial sequential decision-making environments, but practitioners face a design space with three orthogonal dimensions: context representation (what the agent sees), deliberation (how it reasons), and hierarchy (how tasks decompose). Prior work treats these as independent levers to pull—add chain-of-thought for better reasoning, add hierarchy for task decomposition, add state tracking for better context. No one has systematically measured which combinations actually improve performance versus merely burn tokens in a controlled adversarial POMDP setting where failure mitigation is the only goal.

Problem: Compound LLM agents in adversarial POMDPs
   |
   v
Gap: Design choices (context/deliberation/hierarchy) studied in isolation
   |
   v
Assumption: These dimensions interact—combinations may interfere
   |
   v
Method: Controlled ablation study (5 model families, 12 configs, 3475 episodes)
        + Token-level cost accounting in CybORG CAGE-2 cyber defense
   |
   v
Evidence: (1) Programmatic state abstraction: +76% return, best RPTS
          (2) Deliberation cascade: -3.4x return, +2.7x tokens
          (3) Hierarchy alone > Hierarchy + deliberation
   |
   v
Conclusion: In structured adversarial POMDPs, invest in infrastructure
            (state abstraction, clean decomposition) over per-agent reasoning

The Increment

One sentence: Before this paper, practitioners assumed more reasoning and deeper hierarchies would improve compound LLM agents; after, we know that combining deliberation with hierarchy creates destructive interference in adversarial POMDPs, and programmatic state abstraction delivers better returns per token than self-questioning tools.

Core Mechanism

The study manipulates three design dimensions independently and in combination. Context representation varies between raw observations (the agent sees everything the environment emits) and a deterministic state-tracking layer that compresses history into a structured belief state. Deliberation adds self-questioning, self-critique, and self-improvement tools, optionally with chain-of-thought prompting. Hierarchy decomposes the task: a monolithic ReAct agent handles everything, or a coordinator delegates to specialized sub-agents (reconnaissance, analysis, action execution).

All configurations run in CybORG CAGE-2, a cyber defense simulator modeled as a POMDP with non-positive rewards—agents mitigate damage rather than maximize gain. The evaluation spans 3,475 episodes across five model families (GPT-4o, Claude 3.5 Sonnet, Gemini 1.5 Pro, Llama 3.1 405B, Qwen 2.5 72B) with token-level cost accounting. Each configuration’s return per token spent (RPTS) reveals which design choices pay off.

Raw Obs -----> LLM Agent -----> Action
                  ^
                  |
            [Deliberation Tools]
            [Self-Q, Critique, Improve]

vs.

Raw Obs -> State Tracker -> Compressed State -> LLM Agent -> Action
           [Deterministic]                         ^
                                                   |
                                              [Optional Tools]

vs.

Compressed State -> Coordinator LLM
                        |
                        +---> Recon Sub-Agent
                        +---> Analysis Sub-Agent
                        +---> Action Sub-Agent
                              [Each with optional deliberation]

Think of this like designing a restaurant kitchen. Context representation is whether chefs see every ingredient delivery (raw observations) or a prep station that pre-organizes ingredients into mise en place (state abstraction). Deliberation is whether each chef stops to question their own decisions, critique their technique, and revise their plan before acting. Hierarchy is whether one chef handles the entire dish (monolithic) or a head chef delegates to stations (hierarchy).

The study finds that mise en place (state abstraction) is the highest-leverage investment—it reduces cognitive load and improves output quality. Adding deliberation to every station chef (deliberation cascade) creates bottlenecks: each chef second-guesses themselves, coordination overhead explodes, and the kitchen slows down without improving dish quality. The best setup is a well-organized prep station with a clean delegation structure, where each station chef executes their specialized task without overthinking.

Key Concepts

• Deliberation Cascade: When a hierarchical agent system distributes deliberation tools (self-questioning, self-critique, self-improvement) across multiple sub-agents, performance degrades relative to hierarchy alone. Each sub-agent’s reasoning overhead compounds, coordination becomes brittle, and the system uses 1.8-2.7× more tokens while achieving up to 3.4× worse mean return. It’s like a relay race where every runner stops mid-stride to question whether they’re running correctly—the handoffs break down and the team finishes last despite each runner being “more thoughtful.”

• Return Per Token Spent (RPTS): A cost-efficiency metric that divides cumulative episode return by total tokens consumed. In adversarial POMDPs with non-positive rewards, RPTS reveals which design choices mitigate failure most economically. A configuration with -50 mean return using 10K tokens (RPTS = -0.005) is more cost-effective than -40 return using 20K tokens (RPTS = -0.002). This metric prevents the trap of celebrating absolute performance gains that come from brute-force token expenditure.

• Programmatic State Abstraction: A deterministic layer that sits between raw environment observations and the LLM agent, compressing partial observability into a structured belief state. Instead of feeding the agent “host 192.168.1.5 sent TCP SYN to port 22, host 192.168.1.7 sent ICMP echo request…” it provides “3 hosts active, 2 suspicious connections, 1 known compromised.” This is not learned—it’s hand-coded domain logic that exploits the structure of the POMDP. The LLM reasons over the compressed state, not raw sensor data.

Framework Shift

Before (mainstream approach):        After (this paper):

LLM Agent                            State Tracker (deterministic)
    |                                        |
    v                                        v
[Self-Question]                        LLM Coordinator
    |                                   /    |    \
    v                                  /     |     \
[Self-Critique]                   Sub-A  Sub-B  Sub-C
    |                              (no deliberation tools)
    v                                  |     |     |
[Self-Improve]                         v     v     v
    |                                Actions (specialized)
    v
Action

Assumption: More reasoning = better    Finding: Infrastructure + clean
            decisions                           decomposition > per-agent
                                               reasoning depth

From “make each agent think harder” to “give agents better information and clearer responsibilities,” the core shift is that structured environments reward architectural investment over cognitive depth.

Expert Assessment

Problem choice: This is a real gap. The compound LLM agent design space is exploding, and practitioners are combining techniques without understanding their interactions. The choice to study adversarial POMDPs (cyber defense) is smart—it’s a domain where failure is costly and token budgets matter. The non-positive reward structure forces honest accounting; you can’t hide poor design behind inflated scores.

Method maturity: The experimental design is solid. Controlled ablation across five model families with token-level cost accounting is rare and valuable. The deliberation cascade finding is genuinely surprising—most researchers assume more reasoning helps. However, the study is limited to one environment (CybORG CAGE-2). The generalization claim to “structured adversarial POMDPs” is plausible but not proven. A second domain (e.g., adversarial planning, competitive games) would strengthen the case.

Experimental integrity: Baselines are fair. The authors test all combinations rather than cherry-picking. Token accounting is transparent. One concern: the state-tracking layer is hand-coded for CybORG. How much domain expertise does this require? If state abstraction is the winning move but requires expert-level environment modeling, the practical barrier is high. The paper acknowledges this but doesn’t quantify the engineering effort.

Writing quality: The paper is dense but well-structured. The deliberation cascade concept is clearly defined and illustrated. The weakest section is the related work—it reads like a literature dump rather than positioning the contribution. Rewriting it to emphasize the gap (no prior work studies these three dimensions jointly in adversarial POMDPs) would sharpen the narrative. The results section could benefit from a decision tree: “If you have budget for one improvement, do X. If you have budget for two, do X and Y.”

Verdict: weak accept — The deliberation cascade finding is novel and the experimental rigor is above average, but the single-environment limitation and unclear generalization boundary prevent a strong accept. This is a valuable empirical contribution that will inform practitioner decisions, but it’s not a paradigm shift.

Takeaways

1. Invest in state abstraction first: If you’re building LLM agents for structured environments (games, simulations, cyber defense), write a deterministic layer that compresses observations into a belief state. This delivers the highest return per token spent. Don’t feed raw logs to the LLM—give it a dashboard.

2. Hierarchy without deliberation: If you decompose tasks across sub-agents, don’t give each sub-agent self-questioning tools. Let the coordinator reason, let the sub-agents execute. Deliberation cascades are real and expensive.

3. Cost-efficiency metrics matter: Track return per token spent, not just absolute performance. A configuration that scores 10% better but uses 3× more tokens is often a bad trade in production.

4. The “more reasoning = better” heuristic breaks in adversarial POMDPs: Chain-of-thought and self-critique help in knowledge tasks (math, coding, question-answering). In sequential decision-making under partial observability with adversarial dynamics, they can hurt. The environment punishes hesitation and coordination overhead.

5. Transferable design principle: In domains where you can exploit structure (known state space, deterministic transitions, clear task boundaries), invest in programmatic infrastructure over LLM reasoning depth. Save the LLM’s cognitive budget for the irreducibly uncertain parts.