Agon: When Two AI Models Compete, Both Learn to Think Better

Paper: 2607.07690 Authors: Vladislav Beliaev Categories: cs.LG, cs.AI, cs.CL

The Gap

Reinforcement learning from verifiable rewards (GRPO, DeepSeek-R1 style training) has become the default engine for reasoning models. It works beautifully on problems where you can check the final answer — math, code with tests. But it has a well-known blind spot: it only grades the output, never the thinking. On hard problems, this incentivizes models to write longer, more exhaustive traces rather than to think more efficiently. The trace itself — the actual reasoning — is never evaluated.

Researchers have tried to fix this with process reward models (PRMs) that label intermediate steps, but collecting step-level labels is expensive, noisy, and doesn’t generalize. Others have used self-consistency or best-of-N sampling at inference, which helps but doesn’t change what the model learns during training.

Agon asks a deceptively simple question: what if the process reward signal comes not from human labels, but from a competing model? If Model A drafts a solution and Model B reads that draft before answering — and both are rewarded for beating the other — then the quality of reasoning is implicitly graded through competitive outcome. No labels needed. No reward model needed.

Problem: RL trains on final-answer reward only
    |
    v
    | On hard problems, model learns "write more"
    | not "think better" -- traces never graded
    |
    v
Assumption: A rival that reads your draft can
    | expose weak reasoning implicitly
    |
    v
Method: Two models compete, alternating
    | drafter and reader roles each episode
    |
    v
Evidence: 2x pass@1 on DeepMath hard split
    | 8x the gain of untrained Mixture-of-Agents
    |
    v
Conclusion: Implicit competitive grading rivals
    explicit process labels at zero annotation cost

The Increment

One sentence: Before this paper, grading reasoning required explicit step-level supervision or a separate reward model; after this paper, a competing peer that reads your draft *is the reward signal.

Core Mechanism

Agon sets up a competitive self-play loop between two models of comparable strength but different behavioral profiles (e.g., two different Qwen3 checkpoints or Qwen3 vs. Gemma 4). Both models attempt the same problem. In each episode, one model plays the “drafter” role — it writes a full solution — while the other plays the “reader” role — it sees the drafter’s solution and then writes its own answer. Roles alternate: the drafter becomes the reader and vice versa in the next episode.

Each model receives a simple binary reward: did it get the right answer? But the reward is relative. Model A gets +1 if it solves the problem and Model B does not (or if both solve it and A’s solution is preferred). This creates a zero-sum pressure: to earn reward, a model must out-reason a rival that has seen its work. If your draft is sloppy or your reasoning is shallow, the reader will improve on it and beat you. If your draft is genuinely insightful, the reader gains less from seeing it.

Both models are optimized simultaneously (each through its own policy gradient). This is the key insight from game theory: each player faces a progressively stronger opponent. In single-model RL, the “opponent” is a fixed reward function that never gets harder. In Agon, as Model A improves, Model B must also improve to keep winning, and vice versa. This creates an escalating arms race that single-model RL simply cannot provide.

At inference time, the pair deploys exactly as it trained: Model A drafts a solution, Model B reads it and produces its final answer. This two-stage cascade costs one extra forward pass but captures the competitive dynamic that was learned during training.

Agon Training Loop (one episode):
    |
    +---> Problem P sampled from dataset
    |
    +---> Role Assignment (alternates each episode)
    |         |
    |         +---> M1 = DRAFTER
    |         +---> M2 = READER
    |
    +---> M1 generates draft solution S1 for P
    |
    +---> M2 sees (P, S1), generates answer S2
    |
    +---> Verify: S1 correct? S2 correct?
    |         |
    |         +---> r1 = f(S1, S2)  [M1 reward]
    |         +---> r2 = f(S2, S1)  [M2 reward]
    |
    +---> Policy gradient update on M1 (r1)
    |     Policy gradient update on M2 (r2)
    |
    +---> Swap roles for next episode
          M1 = READER, M2 = DRAFTER

Structural Metaphor: The Study Group

Imagine two medical students preparing for a brutal board exam. Every night, they pick the same practice question. Student A solves it first, showing all their work on a whiteboard. Student B watches, then sits down and solves the same problem — with the advantage of having seen A’s approach.

Here’s the twist: the exam score is relative. Whichever student answers correctly (or more elegantly) gets the point. If both get it right, the one whose reasoning was cleaner wins.

Student A can’t afford to be sloppy on the whiteboard — B will spot the gap and do better. But A also can’t afford to be merely correct; the reasoning must be so solid that B can’t easily improve on it. Student B, meanwhile, can’t just copy A’s work — the exam tests understanding, not transcription. B must genuinely learn from A’s draft and then demonstrate superior grasp.

Over weeks of this, both students get dramatically better. Not because a professor is grading their scratch work (no process labels), and not because they’re doing more problems alone (not single-model RL). They improve because they face a peer who sees their thinking and is incentivized to beat it. The quality of their reasoning is implicitly judged every night by whether the other student could exploit its weaknesses.

That’s Agon. The whiteboard is the draft. The board exam is the verification. And the study partner is the competing model.

Key Concepts

• Implicit Process Reward: In traditional RL for reasoning, you need a reward signal. For final answers, this is easy — check if 42 is correct. For reasoning *steps, you need someone to label “this intermediate step is good” or train a separate model to predict that. Agon sidesteps this entirely. The process reward is implicit: if your rival reads your reasoning and then beats you, your reasoning was implicitly rated as “improvable.” If they can’t beat you despite seeing your work, your reasoning was implicitly rated as “robust.” No human ever labels a single step, yet the model gets feedback on the quality of its thinking. It’s like how you don’t need a music teacher to tell you your guitar solo was weak — if the other guitarist in your band plays a better solo every time they hear yours, you know.

• Competitive Self-Play as Curriculum: In single-model RL with a fixed reward function, the difficulty is static. The model solves easy problems, gets reward, and has no incentive to tackle harder reasoning. In Agon, as one model improves, the other must keep up — creating an automatically escalating curriculum. This is the same principle behind AlphaGo’s self-play: the opponent gets harder as you get stronger, so you can never plateau. The curriculum is *endogenous — it emerges from the competition rather than being designed externally.

• Behavioral Diversity as a Prerequisite: Agon doesn’t require two different architectures or wildly different model sizes. It requires models that are comparably strong but *behave differently — they make different mistakes, use different heuristics, have different blind spots. This is what makes the competition productive: each model exposes the other’s unique weaknesses. Two identical copies of the same model would tie forever and learn nothing. The paper finds that different model families (Qwen3 vs. Gemma 4) or even different checkpoints of the same family provide enough diversity.

Framework Shift

Before (mainstream approach):
                                         
  Model --> Generate trace --> Final answer --> Binary reward
    ^                                            |
    |_____________ policy gradient ______________|
                                         
  Problem: reward = 0 or 1 on final answer only
  Long traces != good reasoning, just more coverage
  Process labels needed for trace-level feedback (expensive)

After (this paper):

  Model A --draft--> Model B --answer--> Verify
  Model B --draft--> Model A --answer--> Verify
    ^         ^                |
    |         |    reward = win/lose vs rival
    |_________|________________|
    
  Both optimized simultaneously; rival = moving target
  No process labels; reasoning graded by competitive outcome

From “grading the answer” to “grading the reasoning through competition,” the core shift is that the reward signal itself becomes an emergent property of adversarial dynamics rather than a fixed function.

Expert Assessment

Problem choice: This is a real and well-known gap. The “length bias” in RL-tuned reasoning models — where models learn to write verbose traces instead of thinking sharply — is a genuine pain point for anyone deploying these systems. Process reward models have been the dominant approach to fixing it, but they’re expensive and brittle. The idea of using competition as implicit process supervision is a clean conceptual contribution. It sits at a productive intersection of game theory, multi-agent RL, and LLM alignment.

Method maturity: The core mechanism is clever but not without precedent — self-play has powered breakthroughs from AlphaGo to debate-based AI safety proposals. What’s novel is the specific instantiation: drafter/reader alternation with binary outcome reward on verifiable tasks. The simplicity is a strength. However, the zero-sum framing assumes the problem has a verifiable ground truth (math, code). It’s unclear how this extends to open-ended tasks where “who won” is subjective. The paper acknowledges this but doesn’t address it.

Experimental integrity: The numbers are compelling: 2x pass@1 on DeepMath hard with Qwen3 is a large improvement. The ~8x comparison to untrained Mixture-of-Agents is fair but slightly misleading — MoA without training is a weak baseline, and the real comparison should be against GRPO + best-of-N at matched compute. The cross-family replication (Qwen3, Qwen3.5, Gemma 4) is good practice and adds credibility. One concern: the paper doesn’t clearly separate the effect of competitive dynamics from the effect of simply having two models collaborate (draft → read → answer). An ablation showing that collaborative (non-competitive) two-model training fails would strengthen the claims.

Writing quality: The abstract is excellent — concise, clear about what’s new. The method section could be tighter; the alternating-role mechanism is described in prose but a formal algorithm box or pseudocode would have helped. The comparison to Mixture-of-Agents feels like score-padding rather than essential analysis. The final paragraph about “reasoning in latent space” is a teaser that adds nothing to the paper’s contribution and reads as future-work filler.

Verdict: weak accept — The core idea is clean and the results are strong, but the ablations need tightening and the competitive-vs-collaborative distinction deserves more rigorous treatment.

Takeaways

Three things a practitioner can steal:

1. Competition as free process reward: If you have two models of similar capability and a verifiable task, you can train them against each other instead of building a process reward model. The competitive signal implicitly grades reasoning quality at zero annotation cost. This is especially valuable for domains where step-level labels don’t exist.

2. Deploy the pair as-is: Unlike methods that train with extra infrastructure but deploy a single model, Agon’s inference setup mirrors its training setup — draft then answer. This means you capture the learned competitive dynamic at inference time without additional engineering. The one extra forward pass is a fixed cost.

3. Behavioral diversity > architectural diversity: You don’t need different architectures. You need models that make *different mistakes. If you have two checkpoints of the same model that were trained on different data or with different hyperparameters, that’s likely enough. The key is that each model can expose failure modes the other doesn’t share.