Asymmetric Capacity Allocation: Your Critic Does Not Need to Be Smart

Paper: 2608.21345 Authors: Zhuoyi Yang, Ian G. Harris, Salar Hashemitaheri, Cassie Huang, Yuangang Li, Hyunwoo Oh, Paul Dourish, Tony Givargis, Mohsen Imani, Li Zhang Categories: cs.LG

The Gap

Self-refinement — generate, critique, revise — has become a default mechanism in LLM agent stacks. Ask a model for an answer, ask it to find problems with that answer, ask it to fix them. It is cheap to describe, widely deployed, and usually implemented with one model serving all three roles.

That last part is the problem. Generation, critique and revision are different jobs. Generating requires producing something plausible; critiquing requires recognising what is wrong, which is in some ways an easier discrimination task; revising requires both, plus the ability to act on the criticism. Treating model size as an implementation detail means every stage gets the same budget, regardless of whether it needs it.

The evidence base for that choice is thin. Existing work on self-refinement has not systematically asked how model size affects each stage separately, nor whether effective self-refinement requires equally capable models at all three points. So practitioners allocate capacity uniformly by default — and the paper’s results suggest that default both wastes compute and, in one direction, actively hurts.

   SELF-REFINEMENT PIPELINE
   [generate] -> [critique] -> [revise] -> output
        |            |            |
        |            |            +-- must act on criticism
        |            +-- recognise what is wrong (discrimination)
        +-- produce something plausible (generation)
        |            |            |
        +------------+------------+
                     |
     DEFAULT PRACTICE: one model size for all three
                       (size = "implementation detail")
                     |
                     v
   [GAP] no stage-wise study: does each stage want the
         same capacity? nobody measured it

The Increment

One sentence: Before this paper, self-refinement pipelines spent the same model capacity on generation, critique and revision because nobody had measured whether they needed it; after it, a stage-wise study over 5 benchmarks and 10 model sizes shows the stages scale on different curves, giving a concrete recipe for cutting compute without losing quality.

Core Mechanism

The design is a controlled sweep, and its value comes from breadth rather than novelty. Five benchmarks drawn from different domains, so that a conclusion is not an artifact of one task family. Six model sizes of Qwen3 and four of Gemma 3, so that a conclusion is not an artifact of one family. Then each stage is varied independently, which is the part prior work skipped: instead of testing self-refinement as a monolith, the study asks what happens when the generator, the critic, or the refiner is the one that changes size.

Three findings come out, and they do not point the same way.

Larger generators and refiners generally improve the pipeline, and an undersized refiner can harm it. This is the result with teeth. The refiner is downstream of everything, so a weak refiner does not merely fail to add value — it can take a correct draft and damage it. Capacity spent on the last stage is therefore not a luxury; it is insurance against regression.

Performance is highly insensitive to critic size, but including any critic beats no critique at all. This is the actionable finding. The critic’s job is discrimination — spotting a problem — and that appears to saturate at a small model size. But dropping critique entirely is worse than a tiny critic, so the choice is not “big critic or none”; it is “small critic, and keep it”.

Capacity should not be allocated uniformly. The three stages exhibit distinct scaling characteristics, so a uniform allocation is simultaneously over-provisioning the critic and under-provisioning the refiner — the worst of both directions.

   STAGE-WISE SCALING RESULT

   [generator]  larger  =>  better        (wants capacity)
   [critic]     smaller =>  ~same         (saturates early)
                absent  =>  WORSE         (still needed)
   [refiner]    larger  =>  better
                too small => HURTS the pipeline

   CONSEQUENCE: uniform allocation
     critic  : over-provisioned   (wasted compute)
     refiner : under-provisioned  (active harm)
                |
                v
   RE-ALLOCATE: shrink critic, protect refiner

Think of it as staffing a newspaper’s editorial chain. You want a strong reporter, because the draft is the product. You want a strong copy editor at the end, because a careless editor does not merely fail to improve the piece — they can introduce errors into copy that was already correct, which is the undersized refiner. And you want the fact-checker in the middle, but a fact-checker’s job is narrow and procedural: checking a claim against a source does not improve much with seniority, so a junior who is thorough does nearly as well as a veteran. Spending your senior salary on the fact-checker, and hiring a junior to copy-edit the final draft, is exactly the uniform-allocation mistake this paper measures.

Key Concepts

• Stage-wise scaling: measuring how performance responds to changing the capacity of one stage while holding the others fixed. This is what separates “self-refinement helps” from “this stage of self-refinement helps, and by how much”, and it is the specific gap the paper fills.
• The undersized refiner as a regression source: the finding that a weak final stage can reduce quality below the input draft. It reframes the refiner from an optional polish step into a component whose failure mode is damage, which changes how you budget for it.
• Critique saturation: the observation that critic size barely matters while critique presence matters a lot. This implies the discrimination task is easier than generation, so once a model can spot problems at all, extra parameters add little — but removing the stage loses the benefit entirely.

Framework Shift

Before (uniform allocation):
  [gen: 7B] -> [critic: 7B] -> [refiner: 7B]
  assumption: size is an implementation detail
  outcome: critic over-sized for its job,
           refiner possibly too small to be safe,
           compute spent evenly by default

After (asymmetric allocation):
  [gen: large] -> [critic: small] -> [refiner: large]
  measured: critic saturates early,
            refiner below a threshold causes harm
  outcome: same or better quality,
           less compute on the middle stage

From treating self-refinement as one model doing three jobs, to treating it as three jobs with different capacity requirements, the core shift is recognising that the middle stage is a discrimination task and the last stage carries regression risk.

Expert Assessment

Problem choice: Very good, and usefully unfashionable. The field has spent a lot of effort on whether self-refinement works and comparatively little on how to staff it. Asking an economics question — where does capacity actually pay — about a mechanism everyone already deploys is the kind of paper practitioners can act on immediately.

Method maturity: The method is a grid search, and the paper does not pretend otherwise. That is acceptable because the contribution is empirical, and the breadth (5 benchmarks, 2 model families, 10 sizes) is what gives the conclusions their weight. The weakness is that the study is confined to Qwen3 and Gemma 3; the finding that critics saturate early is plausible for these families and could plausibly shift for a model family with a weaker discrimination ability.

Experimental integrity: Strong on design. Varying one stage at a time is the right control, and using five domains guards against a single-task artifact. The two results that matter most are also the two that could have been embarrassing — that a small refiner hurts, and that critique presence matters more than critique size — so it is to the authors’ credit they are reported plainly rather than smoothed into a general “bigger is better” story.

Writing quality: The three conclusions are stated as separable claims with their conditions attached, which makes the paper easy to use. It would be stronger if it named the threshold: “undersized” is the load-bearing word in the harm finding, and a reader needs to know roughly where the cliff is before they can act on it.

Verdict: accept — an immediately actionable empirical result that replaces a uniform default with a measured allocation, with the caveat that the thresholds remain family-specific.

Takeaways

• Stop sizing every stage of a self-refinement pipeline identically. Put capacity in the generator and the refiner; the critic saturates early.
• Treat the refiner as carrying regression risk, not just upside. A weak final stage can damage a good draft, so it is the wrong place to economise.
• Keep a critic even if it is small. Removing the critique stage costs more than shrinking it, so the trade is “small critic” rather than “no critic”.
• When you evaluate any multi-stage LLM system, vary one stage at a time. The aggregate result hides per-stage scaling behaviour, which is where the compute savings actually live.