Gym-Anything: Scaling Computer-Use Agents to Real Economic Tasks

Paper: 2604.06126 Authors: Pranjal Aggarwal, Graham Neubig, Sean Welleck Categories: cs.LG, cs.AI

The Gap

Existing computer-use agent research focuses on toy tasks: clicking through e-commerce sites, adjusting OS settings, maybe filling out a form. These benchmarks use 5-20 software applications, tasks complete in under 50 steps, and have minimal connection to actual economic activity. The problem isn’t that researchers don’t want harder tasks — it’s that creating environments for complex software (CAD tools, medical imaging systems, enterprise databases) requires weeks of manual setup per application. You need to install dependencies, configure settings, populate realistic data, and verify everything works. This doesn’t scale.

The boundary: we can’t study agents on economically valuable tasks because environment creation is a human bottleneck.

Problem: Manual environment creation doesn't scale
    |
    v
Assumption: Environment setup is itself a task agents can do
    |
    v
Method: Multi-agent pipeline (coder + auditor) automates setup
    |
    v
Evidence: 200 apps -> 10K+ tasks, tasks require 500+ steps
    |
    v
Conclusion: Agents can now train on real economic workflows

The Increment

One sentence: Before this paper, computer-use benchmarks covered toy tasks on ~20 apps; after, we have 10K+ realistic tasks across 200 economically grounded applications.

Core Mechanism

Gym-Anything treats environment creation as a coding task. You give it a software application and a task description. A coding agent writes bash scripts to install the software, download real-world datasets (research papers for reference managers, DICOM files for medical viewers, CAD models for engineering tools), configure settings, and produce evidence that setup succeeded (screenshots, log files, API responses). An audit agent then reviews this evidence against a quality checklist: Is the software actually installed? Is the data realistic? Can the task be completed?

The pipeline runs iteratively. If the auditor finds issues (“the database has no tables” or “the screenshot shows an error dialog”), the coder tries again. Once the auditor approves, the environment is packaged with train/test splits. The authors applied this to 200 applications selected from a taxonomy of U.S. occupations weighted by GDP contribution — so you get medical imaging software, astronomical data analysis tools, enterprise resource planning systems, not just web browsers.

Input: Software + Task Description
    |
    v
+------------------+
| Coding Agent     |  Writes setup scripts
| - Install deps   |  Downloads real data
| - Configure app  |  Produces evidence
+------------------+
    |
    v
Evidence: Screenshots, logs, API responses
    |
    v
+------------------+
| Audit Agent      |  Checks quality
| - Software works?|  Reviews evidence
| - Data realistic?|  Against checklist
+------------------+
    |
    +---> Fail: Loop back to coder
    |
    +---> Pass: Package environment

Think of it like a restaurant opening. The coding agent is the contractor who installs equipment, stocks ingredients, and sets up the kitchen. They take photos and write reports to prove everything’s ready. The audit agent is the health inspector who checks against a list: Is the fridge cold? Are ingredients fresh? Is the stove working? If something’s wrong, the contractor fixes it. Once approved, the restaurant opens for business (the environment is ready for agent training).

The key insight: environment creation has structure. You always need installation, data, configuration, and verification. By framing this as a multi-agent task with explicit evidence and auditing, you can automate what previously required human expertise for each application.

Key Concepts

• Evidence-based setup: Instead of just running scripts and hoping they work, the coding agent must produce artifacts (screenshots showing the app launched, log files confirming data loaded, API calls returning expected results). The auditor reviews these artifacts against a checklist. This mirrors how humans verify complex setups — you don’t trust the script, you check the outcome. It’s the difference between “I ran npm install” and “here’s a screenshot of the dev server running on port 3000.”

• Long-horizon tasks: Most benchmarks have tasks completing in 10-50 steps (click here, type that, submit). CUA-World-Long has tasks requiring 500+ steps — like processing a batch of medical images through multiple analysis stages, or configuring an enterprise system with dozens of interdependent settings. These aren’t artificially lengthened; they reflect real workflows where you navigate complex software, wait for computations, handle errors, and iterate. The challenge shifts from “can the agent click buttons” to “can it maintain context and recover from failures over extended interactions.”

• Economic grounding: The authors built a taxonomy of occupations from U.S. Bureau of Labor Statistics data, weighted by GDP contribution. So instead of picking random software, they selected applications used in high-value occupations: radiologists use medical imaging tools, astronomers use data analysis software, engineers use CAD systems. This ensures the benchmark reflects tasks people actually get paid to do, not just tasks that are easy to automate.

Framework Shift

Before (mainstream approach):        After (this paper):

Human manually creates env           Agent pipeline creates env
    |                                    |
    v                                    v
[Web browser]                        [Coder writes setup]
[Simple e-commerce]                      |
[~20 apps]                               v
[<50 steps/task]                     [Auditor verifies]
                                         |
Limited to what humans                   v
can manually set up                  [200 apps, 10K+ tasks]
                                     [500+ steps/task]
                                     
                                     Scales to any software

One sentence: From human-bottlenecked toy environments to agent-automated realistic workflows, the core shift is treating environment creation itself as an automatable task.

Expert Assessment

Problem choice: This is a real gap. The field has been stuck on WebShop and MiniWoB for years because nobody wants to spend months setting up CAD software environments. The economic grounding is smart — it forces the benchmark toward tasks that matter, not just tasks that are easy to measure. The problem sits at the intersection of scaling (how do we get more environments?) and realism (how do we get tasks that reflect actual work?).

Method maturity: The multi-agent setup is straightforward but effective. The evidence-based auditing is the clever bit — it’s not just “did the script exit 0?” but “can you prove the environment works?” This mirrors how humans actually verify complex setups. However, the paper doesn’t deeply explore failure modes: what happens when the coder and auditor get stuck in a loop? How often does the pipeline produce broken environments that pass auditing? The distillation experiments (2B model outperforming 4B models) suggest the data quality is high, but more ablations on the auditing process would strengthen the claims.

Experimental integrity: The baselines are fair — they compare against existing benchmarks (WebArena, OSWorld) and show CUA-World is harder. The test-time auditing (VLM reviews trajectories and provides feedback) is a nice touch, though the improvement (11.5% → 14.0%) is modest. The 500+ step tasks are genuinely long-horizon, not artificially padded. One concern: the paper doesn’t report inter-annotator agreement on task quality or how many generated environments were discarded. Without this, it’s hard to assess the pipeline’s reliability.

Writing quality: The paper front-loads motivation well but rushes through the technical details. The auditing checklist (Appendix) should be in the main text — it’s central to understanding how quality is enforced. The taxonomy of occupations is mentioned but not visualized; a figure showing GDP-weighted coverage would make the economic grounding more concrete. The distillation experiments feel tacked on; they’d be stronger as a separate contribution or integrated more tightly with the environment creation story.

Verdict: weak accept — Solves a real scaling problem with a practical method, but the evaluation could be more rigorous and the writing could better highlight the core contributions.

Takeaways

Steal the evidence-based verification pattern: When automating complex setups (infrastructure, data pipelines, test environments), don’t just check exit codes — require the automation to produce artifacts (logs, screenshots, API responses) and have a separate process audit them. This catches silent failures and makes debugging tractable.

Use economic grounding to prioritize: If you’re building a benchmark or dataset, weight your selection by real-world value (GDP, user time, market size). This forces you toward tasks that matter and away from tasks that are just easy to measure.

Test-time auditing as a general pattern: The idea of a separate model reviewing completed trajectories and providing feedback is underexplored. It’s not just for computer-use agents — any multi-step generation task (code, proofs, plans) could benefit from a critic that identifies what’s missing or wrong, then feeds that back for refinement.