Teaching AI Agents to Judge, Not Just Imitate: Agentic Critical Training

Paper: 2603.08706 Authors: Weize Liu, Minghui Liu, Sy-Tuyen Ho, Souradip Chakraborty, Xiyao Wang, Furong Huang Categories: cs.AI, cs.CL, cs.LG

The Gap

Current LLM agent training relies on imitation learning - watch an expert, copy their actions. This teaches agents what to do but not why. They never see the contrast between good and bad choices, so they can’t develop judgment about action quality.

Recent work tried to fix this by adding “reflection” - showing agents text that explains why expert actions are better than alternatives. But here’s the problem: the model is still just imitating. It’s copying pre-written reflection text, not learning to reason about quality on its own. It’s like the difference between memorizing someone else’s food reviews versus developing your own palate.

Problem: Imitation learning lacks "why"
    |
    v
Prior fix: Add reflection text to imitate
    |
    v
Limitation: Still imitation, not autonomous reasoning
    |
    v
ACT: Train judgment via RL rewards
    |
    v
Evidence: +5.07 over imitation, +4.62 over RL
    |
    v
Conclusion: Judgment training > reflection imitation

The Increment

One sentence: Before ACT, agents learned by copying actions and reflections; after ACT, agents learn by developing their own judgment through comparing alternatives.

Core Mechanism

ACT presents the model with pairs of actions - one better, one worse - and asks it to judge which is superior. The model outputs its reasoning and choice. If the judgment is correct, it gets rewarded. If wrong, it doesn’t. This is pure reinforcement learning on the judgment task itself.

The training data comes from existing agent trajectories. For each decision point, ACT creates pairs: the expert action versus alternatives (either from other trajectories or generated by the model itself). The model learns to identify quality differences by being rewarded for correct judgments, not by imitating explanation text.

The key insight: by training the model to judge rather than imitate, you force it to develop internal reasoning about what makes actions good or bad. This reasoning capability then transfers to actual agent execution - the model can evaluate its own action candidates and pick better ones.

Training Loop:
    
    [Expert Action] + [Alternative Action]
            |
            v
    Model judges: "Which is better?"
            |
            v
    Model outputs: reasoning + choice
            |
            v
    Reward if correct, no reward if wrong
            |
            v
    Model updates to improve judgment
            |
            v
    Judgment skill transfers to agent execution

Think of it like developing a wine palate. The old way (imitation learning) is reading wine descriptions and memorizing them - “this Bordeaux has notes of blackcurrant and oak.” The reflection approach is reading a sommelier’s comparison notes - “this wine is better because…” But you’re still just memorizing text.

ACT is different: you taste two wines side by side, decide which is better, and get feedback on whether you’re right. Do this enough times, and you develop your own palate - an internal model of what makes wine good. Now when you encounter a new wine, you can evaluate it yourself without needing someone else’s tasting notes. The judgment skill you built transfers to new situations.

In ACT, each component maps directly: the two wines are the action alternatives, your judgment is the model’s reasoning process, the feedback is the RL reward, and your developed palate is the model’s learned reasoning capability. The model isn’t memorizing quality assessments - it’s building the ability to make them.

Key Concepts

• Judgment Training vs Reflection Imitation: Traditional reflection-based methods give the model text like “Action A is better than Action B because X, Y, Z” and train it to generate similar text. This is still supervised learning - the model learns to produce reflection-shaped outputs. Judgment training instead asks “Which is better, A or B?” and rewards correct answers. The model must develop its own reasoning to succeed. It’s the difference between copying someone’s math homework (you learn the answer format) versus solving problems and checking if you got them right (you learn the reasoning process).

• Autonomous Reasoning Development: The paper emphasizes “autonomous” because the model isn’t being told what reasoning to use - it discovers effective reasoning patterns through trial and error. When rewarded for correct judgments, the model experiments with different ways of analyzing actions until it finds approaches that work. This is fundamentally different from distilling reasoning from a teacher model, where the reasoning patterns are predetermined. Autonomous development means the model might discover reasoning strategies the designers didn’t anticipate.

• Judgment Transfer: The surprising part is that training on judgment (comparing actions) improves execution (choosing actions). Why does this work? Because good judgment requires understanding what makes actions effective. When the model learns to identify better actions in comparisons, it builds internal representations of action quality. During execution, these same representations help evaluate candidate actions. It’s like how learning to critique essays makes you a better writer - the analytical skill transfers to production.

Framework Shift

Before (imitation + reflection):        After (ACT):

Expert trajectory                       Expert + alternatives
       |                                       |
       v                                       v
Extract actions + reflection text       Create judgment pairs
       |                                       |
       v                                       v
Model imitates both                     Model judges quality
       |                                       |
       v                                       v
Learns: "copy this pattern"             Learns: "develop reasoning"
       |                                       |
       v                                       v
Execution: recall patterns              Execution: evaluate options

From copying expert behavior and explanations to developing autonomous judgment capability, the core shift is from pattern matching to reasoning development.

Expert Assessment

Problem choice: This is a real gap. The limitation of imitation learning in agent training is well-documented, and the reflection-based fixes do feel like band-aids - you’re still in the supervised learning paradigm. The problem sits at a natural inflection point where the field is moving from “make agents that can follow instructions” to “make agents that can reason about their choices.”

Method maturity: The core idea is elegant - use RL to train judgment rather than imitation to copy reflection. It’s not revolutionary (preference learning and comparison-based training exist), but the application to agent training is clever. I don’t see simpler approaches being overlooked. The method is straightforward enough that it should be reproducible.

Experimental integrity: The baselines look fair - they compare against imitation learning, standard RL, and knowledge distillation approaches. The improvements are consistent across three benchmarks (not cherry-picked). The OOD generalization results are particularly convincing - if it only worked on training distributions, I’d be skeptical. One minor concern: the paper doesn’t deeply analyze failure cases or discuss when ACT doesn’t help.

Writing quality: The abstract and introduction are clear. The method section could use more detail on how they construct the alternative actions and handle the reward signal. The results section is solid but could benefit from more error analysis. If I were revising, I’d expand the “why does judgment transfer to execution” discussion - that’s the key insight and it deserves more theoretical grounding.

Verdict: weak accept to strong accept - This is solid work with a clear contribution, good experimental validation, and practical value. The core insight (judgment training > reflection imitation) is well-executed. Not groundbreaking, but a meaningful step forward in agent training methodology.

Takeaways

For practitioners building LLM agents: Don’t just train on expert demonstrations. Create comparison datasets where your model judges between good and bad actions. This is relatively easy to implement - take your existing trajectory data, pair successful actions with alternatives, and train with preference learning or RL on judgment accuracy.

For researchers in RL/LLM intersection: The judgment-to-execution transfer is worth exploring more deeply. What other capabilities can you develop through comparison training that transfer to generation? Could you train better code generation by training judgment on code quality comparisons?

For anyone doing imitation learning: When you add “reflection” or “reasoning” to your training data, ask: is the model learning to reason or learning to imitate reasoning-shaped text? If it’s the latter, consider whether judgment training could develop more genuine capability.

Specific technique to steal: The alternative action construction method. They use both (1) actions from other trajectories at similar states and (2) model-generated alternatives. This creates a rich comparison dataset without needing human annotation of “why” each action is better.