ClawBench: Can AI Agents Complete Everyday Online Tasks?

Paper: 2604.08523 Authors: Yuxuan Zhang, Yubo Wang, Yipeng Zhu, et al. (UBC, Vector Institute, Etude AI, CMU, Waterloo, and others) Website: claw-bench.com Categories: cs.CL

The Sandbox Illusion

AI agents can navigate graphical interfaces, fill forms, and execute multi-step workflows. Commercial systems like OpenAI Operator and Anthropic Computer Use demonstrate impressive capabilities. Open-source agents like OpenClaw show promise. But can they actually handle the everyday online tasks people depend on—booking flights, ordering groceries, submitting job applications?

We don’t know. Because existing benchmarks don’t test this.

WebArena, VisualWebArena, OSWorld, TheAgentCompany—all evaluate agents in offline sandboxes with static HTML, fixed DOM structures, no authentication, and no dynamic content. This controlled setting simplifies evaluation but removes the very complexities that make real-world web interaction difficult: cookie consent pop-ups, dynamic JavaScript rendering, anti-bot defenses, constantly evolving layouts.

Benchmarks that do operate on real websites—WebVoyager, AssistantBench, Online-Mind2Web, Claw-Eval—are limited to read-only information retrieval or mock APIs for testing simple write operations. Write-heavy task completion on live platforms remains unevaluated.

The result: Models achieve 65-75% success on established benchmarks but we have no reliable picture of how well they can actually “get things done” on the real web.

ClawBench: Real Tasks, Real Websites

ClawBench introduces 153 everyday online tasks spanning 15 life categories across 144 live production platforms:

• Daily Life (21 tasks): Grocery delivery, meal planning, home services
• Shopping (16 tasks): Product purchases, price comparisons, returns
• Entertainment (15 tasks): Movie tickets, event bookings, streaming
• Dev & Tech (15 tasks): API documentation, package management, deployment
• Travel (13 tasks): Flight booking, hotel reservations, car rentals
• Pets (11 tasks): Vet appointments, pet supplies, grooming
• Rating (10 tasks): Review submission, rating products/services
• Education (9 tasks): Course enrollment, assignment submission
• Office (9 tasks): Document management, calendar scheduling
• Job Search (8 tasks): Application submission, resume upload
• Social (8 tasks): Profile updates, connection requests
• Finance (6 tasks): Bill payment, account management
• Academia (5 tasks): Paper submission, citation management
• Personal Mgmt (4 tasks): Health records, personal data
• Automation (3 tasks): Workflow setup, integration configuration

Each task is individually straightforward for a human (typically under 30 minutes) but involves production websites with all their real-world complexity.

Safe Evaluation on Live Sites

Rather than recreating websites in sandboxes, ClawBench lets agents operate on production sites and addresses safety with a targeted mechanism:

Interception Layer: A lightweight Chrome extension records low-level browser actions, paired with a CDP-based instrumentation server that monitors outgoing network traffic and intercepts the final submission request—the single HTTP call that would commit an irreversible transaction.

Five Layers of Behavioral Data:

1. Session replay via Xvfb virtual display and FFmpeg
2. Per-step action screenshots
3. HTTP traffic logs
4. Agent messages (reasoning traces and tool calls)
5. Low-level browser actions (clicks, keystrokes, scrolls)

Human annotators complete every task under the same setup to produce ground-truth trajectories.

Agentic Evaluation

Agent trajectories are scored by an Agentic Evaluator—a Claude Code sub-agent invoked under a fixed evaluation rubric. The evaluator consumes:

• Task instruction
• Human reference actions and payloads
• Agent’s executed actions and payloads

It applies predefined evaluation criteria and produces a binary verdict with structured justification grounded in the request schema and step-level evidence.

Traceable Analysis: The evaluation reveals not just whether the agent failed, but which required fields or steps were incorrect and where the agent diverged from the reference trajectory.

Results: The Reality Gap

Seven frontier models evaluated on ClawBench:

Model	Success Rate
Claude Sonnet 4.6	33.3%
GLM-5	24.2%
Gemini 3 Flash	19.0%
Claude Haiku 4.5	18.3%
GPT-5.4	6.5%
Gemini 3.1 Flash Lite	3.3%
Kimi K2.5	0.7%

The strongest model completes only one-third of everyday tasks. Two of seven models score below 5%.

Comparison to sandbox benchmarks:

• OSWorld (verified): 66.4%
• WebArena (verified): 67.3%
• ClawBench: 33.3% (Claude Sonnet 4.6)

The gap is stark. Sandbox performance does not predict real-world capability.

What Makes Real Tasks Hard?

ClawBench tasks require capabilities beyond existing benchmarks:

1. Document-Grounded Information Extraction Tasks require obtaining relevant information from user-provided documents (resumes, receipts, medical records) and correctly transferring it to web forms.

2. Multi-Step Workflows Across Diverse Platforms Agents must navigate complex interaction patterns that vary across 144 different platforms, each with unique UI conventions, authentication flows, and error handling.

3. Write-Heavy Operations Filling detailed forms correctly—job applications with 20+ fields, travel bookings with passenger information, medical forms with insurance details—requires precision and context retention.

4. Dynamic Content Handling Production websites render content dynamically via JavaScript, present cookie consent pop-ups, implement anti-bot defenses, and change layouts without notice.

5. Real-World Constraints Authentication requirements, session timeouts, rate limiting, CAPTCHA challenges, and other production realities that sandboxes eliminate.

Category Breakdown

Performance varies significantly by task category:

Highest success (Claude Sonnet 4.6):

• Simple form filling: ~45-50%
• Read-heavy tasks: ~40-45%

Lowest success:

• Multi-platform workflows: ~15-20%
• Document-grounded tasks: ~20-25%
• Tasks requiring complex authentication: ~10-15%

The pattern is clear: agents struggle most with tasks that require integrating information across contexts, maintaining state across multiple steps, and handling the unpredictability of production environments.

Implications

For AI agent developers: Sandbox benchmarks are necessary for rapid iteration but insufficient for deployment readiness. ClawBench provides a reality check—a measure of how agents perform on the tasks users actually care about.

For researchers: The 153 tasks and human reference trajectories provide a rich dataset for analyzing failure modes. The agentic evaluator’s traceable verdicts enable fine-grained error analysis: which fields were missed, which steps were skipped, where reasoning broke down.

For users: Current AI agents are not yet reliable general-purpose assistants. They can handle simple, well-structured tasks but struggle with the complexity and unpredictability of everyday online activities.

The Path Forward

Progress on ClawBench requires advances in several areas:

1. Robustness to UI variation: Agents must handle diverse layouts, dynamic content, and unexpected UI changes without brittle selectors or hardcoded assumptions.

2. Long-horizon planning: Multi-step workflows require maintaining goals and context across dozens of actions, with the ability to recover from errors and adapt to feedback.

3. Document understanding: Extracting structured information from unstructured documents (PDFs, images, receipts) and mapping it correctly to form fields.

4. Error recovery: Detecting when something went wrong (wrong page, failed action, unexpected popup) and taking corrective action rather than continuing blindly.

5. Safety and verification: Ensuring agents don’t take irreversible actions without user confirmation, especially for financial transactions or data deletion.

ClawBench brings us closer to AI agents that can function as reliable general-purpose assistants by measuring what matters: can they complete the everyday online tasks that people actually need done?