Claw-Eval: Making Agent Evaluation Actually Trustworthy

Paper: 2604.06132 Authors: Bowen Ye, Rang Li, Qibin Yang, Yuanxin Liu, Linli Yao, Hanglong Lv, Zhihui Xie, Chenxin An, Lei Li, Lingpeng Kong Categories: cs.AI

The Gap

Existing agent benchmarks (WebArena, AgentBench, GAIA) evaluate like grading a math test by only checking the final answer. They miss the work shown. Three critical blindspots: (1) trajectory-opaque grading that ignores how agents reach conclusions, (2) underspecified safety/robustness checks that let dangerous paths slip through, (3) narrow task coverage that doesn’t reflect real-world multimodal complexity.

The problem: when an agent books the right flight but leaks your credit card in the process, current benchmarks call it a success.

Problem: Agent does right thing wrong way
    |
    v
Assumption: Execution path matters as much as outcome
    |
    v
Method: Triple-channel evidence (traces + logs + snapshots)
    |
    v
Evidence: 44% safety violations invisible to output-only eval
    |
    v
Conclusion: Trajectory-aware grading catches what matters

The Increment

One sentence: Before this paper, agent evaluation was pass/fail on final outputs; after, we can grade the entire execution path across completion, safety, and robustness dimensions.

Core Mechanism

Claw-Eval runs 300 tasks across three groups: general service orchestration (booking flights, managing calendars), multimodal perception/generation (processing videos, images, documents), and multi-turn professional dialogue (customer service, technical support). Each task executes three times to separate luck from capability.

The evaluation engine captures three independent evidence streams simultaneously. Execution traces log every API call and function invocation. Audit logs record system-level operations (file access, network requests). Environment snapshots freeze the state before and after each action. These three channels feed into 2,159 fine-grained rubric items that check not just “did it work” but “did it work safely and consistently.”

Scoring happens across three dimensions. Completion measures task success. Safety checks for data leaks, unauthorized access, and harmful outputs. Robustness injects controlled errors (network failures, malformed inputs) to test graceful degradation. The protocol reports Average Score (mean across trials), Pass@k (best of k attempts), and Pass^k (all k attempts succeed) to distinguish models that occasionally get lucky from those with genuine reliability.

Structural metaphor: Think of evaluating a surgeon. Old benchmarks check if the patient survived (final output). Claw-Eval is like having three cameras in the operating room: one filming the surgeon’s hands (execution traces), one monitoring vital signs (audit logs), one recording the surgical field (environment snapshots). The evaluation rubric checks not just survival but whether the surgeon followed sterile protocol (safety), used the right instruments (completion), and handled complications smoothly (robustness). Running three surgeries distinguishes a skilled surgeon from someone who got lucky once.

Key Concepts

• Trajectory-aware grading: Traditional evaluation is like judging a chess game by only seeing the final board position. You miss whether the player blundered into a winning position or executed a brilliant strategy. Trajectory-aware grading watches every move. In agent evaluation, this means recording each API call, each file access, each decision point. When an agent successfully books a flight but accidentally queries the wrong database first, output-only evaluation sees success; trajectory-aware grading sees the security risk. Concrete example: Agent A and Agent B both retrieve the correct document. Agent A directly accesses the authorized file. Agent B tries three unauthorized files first, then succeeds. Same output, radically different safety profiles.

• Pass@k vs Pass^k: These metrics capture different aspects of reliability. Pass@3 asks: “In three attempts, did you succeed at least once?” It measures peak capability—can the model ever get it right? Pass^3 asks: “Did you succeed all three times?” It measures consistency—can you reliably get it right? A model with Pass@3 = 80% but Pass^3 = 40% is like a basketball player who makes spectacular shots but can’t maintain form. The gap between these metrics reveals whether you’re seeing genuine capability or lucky outcomes. Under error injection, Pass@3 stays stable (peak ability unchanged) while Pass^3 drops 24% (consistency collapses).

• Triple-channel evidence: Imagine investigating a crime with three witnesses who can’t collude. Execution traces are the suspect’s own testimony (what the agent reports doing). Audit logs are the security camera (what the system recorded). Environment snapshots are forensic evidence (physical state changes). Each channel catches different violations. An agent might report “successfully sent email” (execution trace) while audit logs show it accessed unauthorized contacts, and environment snapshots reveal it modified files it shouldn’t touch. The 44% of safety violations missed by output-only evaluation? They’re visible in audit logs and snapshots but invisible in execution traces.

Framework Shift

Before (output-only):                After (trajectory-aware):

Task --> [Agent] --> Output          Task --> [Agent] --> Output
          |                                     |||
          v                                     vvv
       Pass/Fail                    Trace + Log + Snapshot
                                              |
                                              v
                                    2,159 rubric items
                                              |
                                              v
                                    Completion + Safety + Robustness
                                              |
                                              v
                                    Avg / Pass@k / Pass^k

From binary outcome judgment to continuous process monitoring, the core shift is treating agent execution as a transparent workflow rather than a black box.

Expert Assessment

Problem choice: This is a real gap, not manufactured. As LLMs move from chatbots to autonomous agents, the evaluation infrastructure hasn’t kept pace. The field is deploying agents in production while using benchmarks designed for question-answering systems. The timing is right—this addresses a genuine need before it becomes a crisis.

Method maturity: The triple-channel evidence approach is clever but not revolutionary. It’s essentially applying software engineering best practices (logging, tracing, state management) to agent evaluation. The insight is recognizing that agent evaluation is closer to system testing than model benchmarking. The Pass@k vs Pass^k distinction is borrowed from code generation (pass@k from Codex evaluation) but the application to consistency measurement is novel. No simpler approach is being overlooked—this is the right level of complexity for the problem.

Experimental integrity: The baselines are fair—14 frontier models including GPT-4, Claude, Gemini. The 44% safety violation miss rate is the paper’s strongest claim, and it’s well-supported by showing specific examples where output-only evaluation fails. One concern: the 300 tasks are “human-verified” but the paper doesn’t detail the verification process or inter-rater reliability. The error injection experiments (controlled network failures, malformed inputs) are well-designed, but the paper could be more transparent about which errors were injected and why those specific failure modes were chosen.

Writing quality: The abstract is dense but clear. The related work section is thorough but could be tightened—it spends too much space on tangential benchmarks. The results section is where the paper shines: the tables are well-organized, the analysis is crisp. The weakest section is the task design methodology (Section 3.2)—it describes what tasks exist but not how they were selected or validated. Rewriting that section with concrete examples and selection criteria would elevate the whole paper.

Verdict: strong accept — This paper addresses a critical gap in agent evaluation with a well-executed solution, strong empirical results, and immediate practical value for the field.

Takeaways

Steal the triple-channel pattern: Any system evaluation can benefit from multiple independent evidence streams. If you’re evaluating RAG systems, don’t just check answer correctness—log retrieval queries, monitor database access patterns, snapshot the knowledge base state. The channels should be independent enough that gaming one doesn’t game the others.

Use Pass@k and Pass^k together: This metric pair transfers to any stochastic system. Evaluating image generators? Pass@5 shows creative potential, Pass^5 shows production reliability. Training RL agents? The gap between these metrics reveals whether you’ve learned a robust policy or found a lucky seed.

Design rubrics before running experiments: The 2,159 fine-grained rubric items weren’t an afterthought—they were defined upfront based on task requirements. This forces you to operationalize “safety” and “robustness” before seeing results, preventing post-hoc rationalization. Concrete technique: for each task, write down 5-10 things that could go wrong, then design checks for each.

Error injection for consistency testing: The controlled error injection (network failures, malformed inputs) is a transferable technique. If you’re building any production system, inject realistic failures during evaluation. The paper shows that peak capability (Pass@k) is robust to errors while consistency (Pass^k) degrades—this distinction matters for deployment decisions.