Beyond Success Rate: How to Evaluate Security AI Agents by What They Actually Cost

Paper: 2607.15263 Authors: Paul Kassianik, Blaine Nelson, Yaron Singer Categories: cs.CR, cs.AI

The Gap

Here’s the state of affairs: every security agent benchmark you’ve seen — Cybench, InterCode-CTF, the whole lot — reports one number: peak success rate. Did the model pop the box? Did it find the vulnerability? Great, write it on the leaderboard. But this assumes the agent gets to burn through as many LLM calls, tool invocations, and telemetry queries as it wants. In a real SOC or pentest engagement, you have a budget. Every reasoning step costs money and time. Every unnecessary tool call is noise.

The problem isn’t that nobody knows this. It’s that nobody has built a systematic framework to measure it. Prior work (Greshake et al.’s offensive agent taxonomies, various CTF leaderboards) treats security tasks as pure capability tests — like judging a race car solely by its top speed while ignoring fuel consumption. This paper fills that gap by introducing cost-aware evaluation that decomposes performance along two axes (inference spend and tool spend) and compares models at fixed cost levels, not just peak performance.

Existing benchmarks        This paper's framing
measure peak success       measures cost-adjusted success
         |                          |
         v                          v
Assumption: more compute    Problem: budget is finite,
always helps                 and tasks scale differently
         |                          |
         v                          v
Method: run until success   Method: decompose cost into
or timeout, report best     inference + tool spend,
                            compare at fixed budget
         |                          |
         v                          v
Evidence: offensive tasks   Evidence: offensive tasks
improve with compute        scale with compute, defensive
(prior work confirms)       tasks do NOT -- tool
                            discipline matters more
         |                          |
         v                          v
Conclusion: agent =         Conclusion: evaluate agents
smart model                 as operational systems with
                            economic constraints

The Increment

One sentence: Before this paper, we judged security agents by whether they could succeed at all; after this paper, we can judge them by how much it costs to succeed — and we’ve learned that offense and defense have fundamentally different cost structures.

Core Mechanism

The evaluation framework operates along two independent cost dimensions. Inference cost captures the LLM compute consumed: number of tokens generated, number of model calls, and the price tier of the model being used. Tool cost captures everything the agent does in the environment: API calls to security tools, telemetry queries to SIEMs, enrichment requests to threat intelligence feeds, and any other external system interaction. By tracking both independently, you can see where the budget is actually going.

The experimental design is straightforward but disciplined. They take two task families — offensive Cybench CTF challenges and defensive Splunk BOTS v1 SOC investigation challenges — and run multiple frontier and open-weight models through them. Instead of reporting best-of-N success, they compute success rate at specific cost thresholds (e.g., “at $0.50 of total spend, what’s the solve rate?”). This produces cost-success curves rather than single-point numbers.

The critical insight comes from comparing these curves across task types. For CTF challenges, the curves show classic scaling behavior: more budget consistently yields more solves, and open-weight models can approach proprietary ones if given equivalent compute. For SOC investigations, the curves flatten or even regress with more budget — the agent gets lost in telemetry, makes redundant queries, or spirals into unproductive reasoning chains. Success here correlates with efficient tool use, not raw compute.

Components and Data Flow:

  [Agent (LLM)] <---inference tokens---> [Cost Tracker]
       |                                        |
       | tool calls                     inference $
       v                                        |
  [Security Tools] <---api calls--------> [Cost Tracker]
       |                                        |
       v                                        v
  [Task Result]                         [Budget Ledger]
       |                                        |
       v                                        v
  [Success/Fail] ----+---------> [Cost-Success Curve]
                     |
  [Cost Ledger] -----+

  Budget Ledger decomposes into:
    inference_spend = tokens * price_per_token
    tool_spend = n_tool_calls * avg_tool_cost
    total_spend = inference_spend + tool_spend

Here’s a structural metaphor that makes this click: imagine you’re evaluating two kinds of delivery drivers. The first drives highway routes (offense/CTF): give them more fuel budget, and they’ll reach more destinations. Faster car + more fuel = more deliveries. The second drives downtown routes (defense/SOC): give them more fuel, and they might just circle the block more times. What matters downtown is knowing the streets, making the right turns, and not wasting time on dead ends. The “cost-aware evaluation” is like switching from “who can deliver the most packages with unlimited gas?” to “who delivers the most packages per dollar of gas, and what does the efficiency curve look like at different budget levels?” The highway driver and the downtown driver have completely different efficiency profiles — and you’d never see that if you only measured peak deliveries.

Key Concepts

• Cost-Success Curve: Think of it like a restaurant’s profit chart. You could plot “how much money did we spend on ingredients” on the x-axis and “how many dishes did we successfully sell” on the y-axis. A great chef’s curve rises steeply — small ingredient spend yields lots of sold dishes. A wasteful chef’s curve is flat — they burn through ingredients without selling more. This paper plots exactly this for security agents: money spent vs. challenges solved. The shape of the curve tells you everything: steep means efficient, flat means the agent is just burning money.

• Inference Spend vs. Tool Spend: This is the distinction between “thinking cost” and “doing cost.” When a security analyst sits at their desk thinking about what to investigate next, that’s inference cost (the LLM’s brain time). When they actually open Splunk, run a query, check VirusTotal, or pull network logs, that’s tool cost (real-world actions). Most benchmarks conflate these. This paper separates them because, in practice, you might want to optimize one without touching the other — maybe your LLM calls are cheap but your SIEM API charges per query.

• Scaling Regimes: Not all tasks respond to “throw more compute at it” the same way. For CTF challenges, the relationship between compute and success is roughly monotonically increasing — more thinking = more wins. For SOC investigations, the relationship is non-monotonic or plateaus quickly. This matters enormously when you’re deciding how to allocate budget. The paper’s key contribution is demonstrating that these regimes exist and are task-type-specific, not model-specific.

Framework Shift

Before (mainstream approach):          After (this paper):

  Task --> Agent --> Pass/Fail         Task --> Agent --> Pass/Fail
                       |                                |
                       v                                v
                  Success Rate                    Cost-Success Curve
                       |                                |
                       v                                v
               Leaderboard                     Budget Decomposition
              (one number)                  (inference $ + tool $)
                                                  |
                                                  v
                                           Scaling Regime Analysis
                                           (offense vs. defense)

From “did it solve the task?” to “how much did it cost to solve the task, and where did the money go?” — the core shift is treating security agents as operational systems with economic constraints, not as academic benchmarks with binary outcomes.

Expert Assessment

Problem choice: This is a genuine gap. Every security agent paper I’ve read reports success rate under generous budgets, and the operational irrelevance of that approach has been an open complaint in the applied security community for at least two years. The insight that offense and defense scale differently isn’t entirely surprising to practitioners, but formalizing it with controlled experiments and cost curves is new and useful. The paper sits at the right moment — as LLM agents move from demos to production, cost-awareness becomes essential.

Method maturity: The framework is conceptually clean but somewhat brute-force in execution. Decomposing cost into inference vs. tool spend is the right granularity for a first pass, but there’s room for more nuance (e.g., latency cost, opportunity cost of parallel vs. sequential tool calls). The use of existing benchmarks (Cybench, Splunk BOTS v1) is pragmatic rather than innovative — they didn’t build new tasks, which is fine for a measurement paper but limits novelty. I’d have liked to see more exploration of why defensive tasks don’t scale: is it context window exhaustion, is it poor tool-use training data, is it the inherently open-ended nature of investigation? The paper gestures at these explanations but doesn’t isolate them.

Experimental integrity: The baselines are reasonable — they compare frontier proprietary models (GPT-4 class) against open-weight alternatives at matched cost levels. The cost tracking methodology is described clearly enough to reproduce. One concern: Splunk BOTS v1 is a single dataset with limited challenge diversity, so the “defensive tasks don’t scale” finding might not generalize to real SOC workflows. They acknowledge this but it’s still a caveat worth flagging. The Cybench evaluation is more robust given the dataset’s size and variety.

Writing quality: The paper is well-structured and readable. The cost-success curve framing is explained clearly. However, Section 5 (related work) feels perfunctory — they cite the major prior work but don’t deeply engage with how their framework relates to adjacent efforts in ML efficiency evaluation (e.g., Pareto-optimal frontier analysis from AutoML). The discussion of “operational fit” in the conclusion is the most interesting part of the paper and deserved its own section with concrete examples.

Verdict: weak accept — The cost-aware framing is the right idea at the right time, and the offense/defense scaling contrast is a genuine finding, but the experimental scope (especially on the defensive side) limits how far the conclusions can reach.

Takeaways

Three concrete things you can steal from this paper:

1. Stop reporting single-number success rates for agent evaluations. If you’re benchmarking any LLM agent (not just security), plot cost-success curves. Compare models at fixed budget levels, not just peak performance. This is immediately applicable to any agent benchmark — coding assistants, research agents, customer service bots.

2. Decompose cost into reasoning cost and action cost. Track LLM tokens separately from tool/API calls. You’ll discover that different tasks have different cost bottlenecks, and this changes which model architecture or prompting strategy is optimal. For SOC-type tasks, optimizing tool-use patterns matters more than upgrading the model.

3. The “scaling regime” concept transfers everywhere. For any agent task family, ask: does success scale monotonically with compute, or does it plateau/regress? This determines whether your strategy should be “bigger model, more compute” or “smarter tool use, better scaffolding.” The paper shows this isn’t just theoretical — it’s empirically measurable and operationally decisive.