From Model Scaling to System Scaling: Why Agent Architecture Matters More Than Model Size

Paper: 2605.26112 Authors: Shangding Gu Categories: cs.AI, cs.LG

The Gap

Existing agentic AI research treats foundation models as the primary object of study. Agents are evaluated on final-task success (did it book the flight? did it fix the bug?), while memory systems, retrieval mechanisms, tool orchestration, and verification layers are treated as implementation details. This model-centric framing worked when agents were simple wrappers around LLMs, but breaks down as agents become long-running systems that maintain state, coordinate multiple tools, and operate under safety constraints.

The gap: we have no systematic way to evaluate or optimize the structured execution layer—the harness—that translates model capability into agent behavior. When an agent fails, we don’t know if the model was too weak, the memory was corrupted, the context constructor dropped critical information, or the orchestration loop made poor routing decisions. The field lacks harness-level benchmarks and design principles.

Problem: Agent failures are opaque
   |
   v
Assumption: Performance emerges from model + harness interaction
   |
   v
Method: Decompose harness into measurable components
        (context governance, memory, skill routing, orchestration)
   |
   v
Evidence: CheetahClaws reference implementation + comparison
   |
   v
Conclusion: System design is now the bottleneck, not model size

The Increment

One sentence: Before this paper, agents were black boxes evaluated on task success; after, the harness is a first-class object with measurable components and design principles.

Core Mechanism

The paper decomposes the agent harness into five layers: (1) foundation model, (2) memory substrate (stores past interactions, learned skills, user preferences), (3) context constructor (selects what goes into the prompt), (4) skill-routing layer (decides which tool or sub-agent to invoke), and (5) orchestration + governance (coordinates execution and enforces constraints). Each layer has failure modes that compound: stale memory poisons context, poor context leads to wrong tool selection, weak orchestration allows unsafe actions.

The method is diagnostic, not algorithmic. Instead of proposing a new agent architecture, the paper provides a framework for measuring harness quality: trajectory coherence (does the agent’s action sequence make sense?), memory hygiene (is stored information accurate and relevant?), context efficiency (is the prompt bloated or missing key facts?), communication fidelity (do tool calls match intent?), verification cost (how much overhead for safety checks?), and safe evolution (does the agent improve without catastrophic forgetting?).

CheetahClaws is a reference implementation that makes these layers explicit and auditable. Unlike monolithic agent frameworks (LangChain, AutoGPT), it separates concerns: memory is a pluggable module, context construction is a distinct step with logging, skill routing has explicit decision traces, and orchestration enforces governance rules before execution.

User Request
     |
     v
[ Context Constructor ] <--- reads --- [ Memory Substrate ]
     |                                        ^
     | (prompt)                               |
     v                                        | (writes)
[ Foundation Model ]                          |
     |                                        |
     | (tool call intent)                     |
     v                                        |
[ Skill Router ] -----> [ Tool Execution ] ---+
     |                         |
     v                         v
[ Orchestration Loop ] <-- [ Governance Layer ]
     |                         |
     +-------------------------+
               |
               v
          Response

Think of the harness as a restaurant kitchen. The foundation model is the head chef—skilled but needs prep work done. Memory is the pantry: if ingredients are spoiled or mislabeled, even a great chef makes bad food. The context constructor is the sous chef who decides what ingredients to bring to the chef’s station—too much clutter and the chef is overwhelmed, too little and the dish is incomplete. Skill routing is the expeditor who decides which station (grill, sauté, pastry) handles each order. Orchestration is the kitchen manager who coordinates timing and enforces health codes. When a dish fails, you can’t just blame the head chef—you need to audit the entire kitchen workflow.

Key Concepts

• Context Governance: The process of deciding what information enters the foundation model’s prompt window. Unlike retrieval-augmented generation (RAG), which focuses on fetching relevant documents, context governance is about budget allocation under constraints. You have 8k tokens. Do you include the full conversation history? Recent tool outputs? User preferences? Relevant code snippets? The wrong choice cascades: if you omit a critical constraint, the model generates an unsafe action; if you include too much noise, the model loses focus. Context governance is the difference between a focused prompt and a junk drawer.

• Trustworthy Memory: Memory that is accurate, relevant, and auditable. Most agent frameworks treat memory as append-only logs or vector databases with no hygiene. This paper argues memory needs three properties: (1) provenance (where did this fact come from?), (2) decay (old information should fade unless reinforced), and (3) verification (can we check if stored beliefs are still true?). Without these, agents accumulate false beliefs and outdated skills. Trustworthy memory is not just storage—it’s a maintenance problem.

• Dynamic Skill Routing: The decision process for which tool, API, or sub-agent handles a given subtask. Static routing (if query contains “weather”, call weather API) breaks down when tasks are ambiguous or require chaining multiple tools. Dynamic routing uses the model’s reasoning to select tools, but introduces new failure modes: the model might hallucinate tool names, misunderstand tool signatures, or route to the wrong abstraction level (calling a low-level API when a high-level workflow exists). Skill routing is where intent meets execution, and mismatches here cause silent failures.

Framework Shift

Before (model-centric):              After (harness-centric):

    [ Big Model ]                    [ Foundation Model ]
         |                                    |
         v                           +--------+--------+
    Task Success?                    |                 |
    (yes/no)                    [ Memory ]      [ Context ]
                                     |                 |
                                     +--------+--------+
                                              |
                                         [ Router ]
                                              |
                                         [ Orchestrator ]
                                              |
                                    +----+----+----+----+
                                    |    |    |    |    |
                              Trajectory  Memory  Context  Verification
                              Quality    Hygiene  Efficiency  Cost

From “did the agent succeed?” to “how did the agent succeed, and can we trust the process?”—the core shift is treating execution infrastructure as measurable and improvable, not just plumbing.

Expert Assessment

Problem choice: Real gap. The field has hit diminishing returns on model scaling for agentic tasks—GPT-4 and Claude 3.5 have similar agent performance despite different model sizes. The bottleneck has shifted to system design, but most research still focuses on model capabilities. This paper names the elephant in the room.

Method maturity: Diagnostic framework, not a novel algorithm. The contribution is conceptual: decomposing the harness into measurable components and arguing for harness-level benchmarks. CheetahClaws is a proof-of-concept, not a production system. The paper would be stronger with empirical evidence that harness improvements outpace model improvements on fixed tasks, but the framing alone is valuable.

Experimental integrity: No controlled experiments—this is a position paper with a reference implementation. The comparison with Claude Code and OpenClaw is qualitative (modularity, auditability) rather than quantitative (task success rates, latency). Fair for a framing paper, but leaves the hard measurement work to future research.

Writing quality: Clear thesis, but the paper tries to cover too much ground. The section on governance mechanisms feels tacked on—either expand it into a full treatment or cut it. The CheetahClaws description is too high-level; showing a concrete failure mode and how the harness design makes it debuggable would be more convincing than architectural diagrams.

Verdict: weak accept — Important framing that shifts research priorities, but needs empirical validation and tighter scope to be a strong contribution.

Takeaways

For practitioners: If you’re building agents, instrument your harness. Log context construction decisions, track memory read/write patterns, and record skill routing choices. When your agent fails, you need to know which layer broke. Treat memory as a database with schema and integrity constraints, not a dump of past interactions.

For researchers: Stop evaluating agents only on final-task success. Measure trajectory quality (did the agent take a reasonable path?), context efficiency (how much of the prompt was actually used?), and memory hygiene (how much stored information is stale?). These metrics reveal optimization opportunities that task success hides.

Transferable idea: The “harness as first-class object” framing applies beyond AI agents. Any system with a powerful but opaque core (compiler, database query planner, recommendation engine) benefits from making the surrounding infrastructure—caching, routing, verification—explicit and measurable. When the core is a black box, optimize the harness.