OmegaUse-OfficeVal: Benchmarking LLM Agents on Office Tasks with Real-World Economics

Paper: 2607.27155 Authors: Jingbo Zhou, Yusai Zhao, Qi Bao, Jingjia Cao, Zhenghai Chen, Chang Gao, Kaiqi Guo, Muxin Guo, Mingxuan Li, Xinjiang Lu Categories: cs.AI, cs.CL, cs.HC

The Gap

Existing agent benchmarks — WebArena, OSWorld, GAIA, SWE-bench — have pushed hard on task complexity and environment realism. But they share a blind spot: none of them ask whether completing a task with an LLM agent is economically rational. They measure “did it work?” but not “was it worth it?” Meanwhile, real office work is multi-step, multi-application, and expensive — a single workflow might take a skilled worker hours. If we’re going to trust LLM agents with office labor, we need to know not just whether they can do it, but whether the quality justifies replacing (or augmenting) a human who costs $X per hour.

The path from gap to contribution:

LLM agent benchmarks measure task success in isolation
  |                       no cost grounding, short-horizon tasks
  v
Gap: Can agents handle REAL office workflows
     at acceptable quality-to-cost ratios?
  |
  v
Method: OmegaUse-OfficeVal
  - 100 tasks from practitioners (~2.32h avg human time)
  - Dual economic signals (labor time + price proxy)
  - Code-based verifiers from fine-grained rubrics
  |
  v
Evidence: Frontier LLMs evaluated against human baseline
  - LLMs are 10-100x cheaper and faster
  - But deliverable quality is far below human level
  |
  v
Conclusion: Cost advantage is real but premature;
            quality gap is the binding constraint

The Increment

One sentence: Before this paper, evaluating LLM agents on office tasks meant checking correctness in a vacuum; after this paper, we have a framework that attaches a dollar sign to both sides of the ledger — human labor cost vs. agent cost — so you can see exactly where the quality-to-cost tradeoff breaks.

Core Mechanism

The benchmark has three layers that work together. First, the task layer: 100 office-suite tasks sourced from real practitioner requests, then sanitized through a privacy-preserving adaptation process. These aren’t synthetic puzzles — they’re the kind of things a knowledge worker actually does (drafting documents, manipulating spreadsheets, preparing presentations). On average, each task takes a human about 2.32 hours.

Second, the economic annotation layer: every task carries two economic signals. One is human labor time (how long a skilled worker takes). The other is a task price proxy (a monetary estimate of the task’s market cost). These aren’t afterthoughts — they’re first-class metadata that enable value-weighted evaluation. A task that takes 4 hours and costs $200 is weighted differently from one that takes 20 minutes and costs$15.

Third, the evaluation layer: fine-grained rubrics define what a correct deliverable looks like for each task. From these rubrics, the authors build code-based verifiers — automated checkers that can assess agent outputs without human judgment calls. This is crucial for reproducibility and for scaling evaluation beyond a handful of tasks.

Practitioner task requests
        |
        v
Privacy-preserving adaptation
        |
        v
+---------------------------+
|  100 office-suite tasks   |<--- Economic signals
|  (avg 2.32h human time)   |     (labor time, price)
+---------------------------+
        |
        v
LLM agent attempts task
        |
        v
Output + Fine-grained rubric
        |
        v
Code-based verifier scores
        |
        v
Quality score + Cost comparison
(human cost vs agent inference cost)

Think of it like a tax audit for AI labor. Imagine your company currently hires a temp worker for each office task. The audit asks: for each line item on the invoice, (1) how much did the human cost, (2) how much would the AI cost, and (3) is the AI’s output actually usable? The “line items” are the 100 tasks. The “human cost” is the labor time and price proxy. The “AI cost” is inference compute. And the “usability” check is the code-based verifier — it’s the auditor who opens each deliverable and checks it against a rubric, not with vibes but with automated criteria. The punchline of the audit: the AI is astonishingly cheap per task, but a disturbing number of its deliverables would get sent back for rework.

Key Concepts

• Economic grounding: Most benchmarks ask “did the agent succeed?” This paper asks “did the agent succeed *relative to what it would cost a human?” Concretely: if a task costs a human $100 in labor and the LLM does it for$0.50 in compute but only delivers 60% quality, is that a good deal? Economic grounding means every task carries explicit cost metadata so you can make that calculation. Without it, you’re comparing agent performance against an imaginary yardstick.

• Value-weighted evaluation: Not all office tasks matter equally. Formatting a slide deck is different from drafting a legal contract. Value-weighted evaluation means that when you compute an overall performance score, each task’s contribution is scaled by its economic weight. A task priced at $200 in human labor counts more than one priced at$10. This prevents the benchmark from being dominated by easy, low-stakes tasks that inflate average scores.

• Code-based verifiers: Instead of asking humans to rate outputs (expensive, slow, inconsistent), the authors build automated checkers from rubrics. A rubric might say “the spreadsheet must contain a pivot table with columns X, Y, Z and values summing to W.” The verifier parses the output and checks each criterion programmatically. This makes evaluation reproducible, cheap, and deterministic — the same output always gets the same score.

Framework Shift

Before (mainstream approach):           After (this paper):
                                       
Task success in isolation              Task success in economic context
  |                                      |
  v                                      v
Accuracy / pass rate as metric         Quality per dollar as metric
  |                                      |
  v                                      v
"How often does it work?"              "Is it worth replacing a human?"
  |                                      |
  v                                      v
Short tasks, synthetic envs            Long-horizon tasks, real workflows
  |                                      |
  v                                      v
Human or LLM-as-judge scoring          Code-based verifiers from rubrics

From “can the agent do it?” to “should the agent do it?” — the core shift is treating LLM agent evaluation as an economic decision, not just a technical one.

Expert Assessment

Problem choice: This is a genuine gap. The field has been so focused on “can agents do X?” that we’ve collectively ignored “at what cost and quality?” The economic framing is overdue, especially as companies start deploying agents in production. Whether 100 tasks is enough to make the economics conclusions robust is debatable, but the conceptual contribution — attaching cost metadata to benchmarks — is valuable regardless of scale.

Method maturity: The economic grounding idea is a clever insight, not brute force. The dual-signal annotation (time + price) is well-designed. However, the paper is somewhat vague on how the “price proxy” is derived — is it based on freelance market rates, salary amortization, or something else? The code-based verifiers are the right approach for reproducibility, but the paper doesn’t deeply discuss failure modes of the verifiers themselves (what if an output is correct but doesn’t match the expected format?).

Experimental integrity: The human baseline is a strength — many agent benchmarks lack this. The finding that LLMs are cheap but low-quality is plausible and consistent with field experience. One concern: the paper doesn’t deeply discuss what “approaching human-level quality” would actually look like numerically. The gap is described qualitatively, but I’d want to see, for instance, what percentage of tasks each model’s output would be “acceptable without revision” by a real office worker. The 10-100x cost advantage claim is eye-catching but would benefit from sensitivity analysis on inference pricing.

Writing quality: The abstract and introduction are clear. The task construction section could be stronger — the privacy-preserving adaptation process deserves more detail, since the validity of the benchmark hinges on whether adapted tasks still look like real office work. The evaluation section reads well but the analysis of *why models fail (beyond “quality gap”) is thin. A failure taxonomy would elevate the paper significantly.

Verdict: weak accept — The economic grounding framework is a genuinely useful contribution that the field needs, but the benchmark’s scale (100 tasks) and some under-specified details (price proxy methodology, verifier edge cases) hold it back from a strong accept. Worth reading and building on.

Takeaways

The stealable idea here is economic grounding for any AI evaluation. If you’re benchmarking an AI system in any domain where humans currently do the work, attach a cost to each task. This immediately reframes the question from “is the AI good?” to “is the AI good *enough for the price?” This framing transfers to coding agents (vs. developer time), customer support bots (vs. agent salaries), medical AI (vs. specialist consults), and more.

The code-based verifier pattern is also worth stealing: define your rubric as machine-checkable criteria upfront, not as human vibes after the fact. This makes any evaluation pipeline cheaper and more reproducible.

Finally, the value-weighting idea is practical wisdom: if you’re evaluating AI on a portfolio of tasks, weight by business impact, not by count. A system that nails 90% of trivial tasks but fails on the 10% that matter most is not a 90% system.