Agent in a Bottle: Can LLM Agents Turn Their Capabilities Into Cheap, Scalable Artifacts?

Paper: 2610.08775 Authors: Ankit Sonthalia, Haritz Puerto, Alexander Rubinstein, Martin Gubri, Seong Joon Oh Categories: cs.AI

The Gap

Frontier large language models can resolve intricate semantic classifications, entity extractions, and data labeling queries with remarkable accuracy. However, modern enterprise workloads routinely process hundreds of millions of instances—such as e-commerce catalog categorization, content moderation, or ad-query matching.

Querying a flagship model (such as Claude 3.5 Sonnet or GPT-4o) individually for every single query instance is economically prohibitive. In practice, human machine learning engineers do not leave high-end models in the loop forever; instead, they invest an upfront engineering budget:

1. Label a gold-standard subset of data.
2. Train a compact, lightweight student model (e.g., DeBERTa or modern SLMs) or synthesize deterministic rule-based programs.
3. Deploy the amortized artifact to process the entire multi-million workload at near-zero marginal cost.

Can autonomous LLM agents perform this meta-engineering task themselves? Until now, AI benchmarks have only evaluated agents on direct task execution, ignoring whether they possess the meta-capability to compile and amortize their intelligence into cheap, scalable artifacts.

   PROBLEM: HIGH-VOLUME WORKLOAD INFERENCE COST COLLAPSE

   10 Million Repetitive Workload Instances
                       |
        +--------------+--------------+
        |                             |
        v                             v
   Direct Zero-Shot Inference    The Human ML Engineering Habit:
   - Cost: $500,000+ API bills   - Invest 3 days upfront
   - Latency: High & unscalable  - Distill small model or write fast script
                                 - Amortized cost drops 500x
                                      |
                                      v
   RESEARCH QUESTION: CAN AUTONOMOUS AGENTS "BOTTLE" THEMSELVES?
   Give Agent: Unlabeled dataset + Fixed time & compute & API budget
   Expectation: Autonomous strategy selection (distillation, code, pruning)
                                      |
                                      v
   EVIDENCE (BOTTLED Benchmark across 10 Models & 3 Tasks):
   - 48 of 60 runs fall below zero-shot confidence interval
   - 31 of 60 runs lose to standard naive distillation baselines
   - Successful runs yield staggering savings: Opus 5 retains 82% F1 at 657x lower cost!
                                      |
                                      v
   CONCLUSION: Zero-shot intelligence != Capability-amortization capability

The Increment

One sentence: Introducing the BOTTLED benchmark, this paper operationalizes and evaluates “bottling”—the ability of LLM agents to autonomously invest an upfront budget to compress their reasoning into cheap, reusable artifacts—demonstrating that while current agents struggle with meta-engineering autonomy, successful bottling delivers up to a 657x reduction in inference costs.

Core Mechanism

The authors formulate the BOTTLED framework across three high-volume production benchmarks (including query-product relevance classification and claim verification):

1. The Bottling Environment:
   • The agent is presented with an entire unlabeled workload ($D_{\text{pool}}$) and task guidelines.
   • The agent is allocated a strictly capped budget: finite wall-clock execution time, GPU compute, and LLM API tokens.
   • The agent is given full autonomy in its action space: it can write Python code, pseudo-label batches of data, train a small student architecture via standard fine-tuning pipelines, construct feature heuristics, or synthesize regex decision rules.
2. The Amortization Objective:
   • The final artifact generated by the agent is frozen and evaluated over the held-out test distribution.
   • The system scores both task performance (macro-F1 / accuracy) and the total amortized cost of processing the entire dataset.
3. Key Findings:
   • The Zero-Shot Disconnect: High zero-shot capability does not imply good bottling capability. Models with near-identical zero-shot accuracy diverge widely when asked to organize their own distillation or scripting workflows.
   • Common Failure Modes: Agents frequently misallocate their token budgets—either over-indexing on labeling uninformative easy samples, introducing silent data leakage, or writing brittle neural network training loops that crash midway.
   • The Peak Upside: When top-tier agents (like Opus 5) execute disciplined bottling protocols, they produce compact classifiers that retain 82% to 94% of top-tier performance while delivering a 657x drop in total financial expenditure.

   THE BOTTLING LIFECYCLE IN BOTTLED

   Unlabeled Workload + Fixed Token & Compute Budget
                          |
                          v
   +----------------------------------------------+
   | Autonomous Agent Meta-Engineering Loop      |
   | 1. Strategic Decision (Code vs Model Distill)|
   | 2. Active Curation & Gold Pseudo-Labeling    |
   | 3. Code Generation & Training Execution      |
   +----------------------------------------------+
                          |
                          v
               Executable Artifact (SLM / Script)
                          |
                          v
   Massive Production Deployment (10M Items)
   -> 657x Lower Running Cost, Low Latency, Zero API Dependence

The load-bearing structural metaphor is a master horologist commissioned to produce one million kitchen timers.

• Direct zero-shot LLM calling is the master horologist personally hand-carving every single one of the one million timers with a jeweler’s loupe and tweezers. The timers are exquisite, but each costs $2,000 and the order takes fifty years to fulfill.
• “Bottling” is the master horologist walking into the machine shop for the first three days, machining a heavy steel stamping die and assembly jig, and teaching an apprentice how to operate the stamping press.
• Many otherwise brilliant horologists fail at this: they either spend all their budget admiring one gear, or build a flimsy wooden jig that snaps on the third timer. But when built right, the apprentice stamps out a million flawless timers for fifteen cents apiece.

Key Concepts

• Bottling (Capability Amortization): The meta-reasoning skill whereby an autonomous agent invests upfront computational and monetary capital to compile its capabilities into a compact, low-cost downstream artifact.
• BOTTLED Benchmark: A standardized test harness assessing how effectively an LLM agent leverages code generation, dataset sampling, and local model training under strict budgetary constraints.
• Zero-Shot Transfer Gap: The quantitative divergence between a model’s raw generative capability and its ability to act as an effective teacher and meta-engineer for downstream artifacts.

Framework Shift

Before (Standard Agent Benchmarks):
  Input: Single Prompt -> Output: Final Answer
  Metric: Zero-Shot Pass Rate / Token Efficiency per instance
  Assumption: Better zero-shot reasoning = Better overall AI capability.

After (BOTTLED Capability Compilation Benchmark):
  Input: Unlabeled Workload + Upfront Budget -> Output: Reusable Software/Model Artifact
  Metric: Amortized Quality vs Total Fleet Cost over 10M samples.
  Reality Check: High zero-shot scores collapse unless the model knows how to engineer scalable systems.

From “treating LLMs as expensive per-query engines forever,” the core shift is that the highest form of agent intelligence is the ability to write itself out of the production runtime loop by synthesizing cheap, permanent artifacts.

Expert Assessment

Problem choice: Visionary and economically grounded. The current AI deployment model—sending billions of everyday business queries to frontier LLM APIs—is financially unsustainable. Evaluating agents on their ability to build cheap software artifacts mirrors actual industrial software engineering.

Method maturity: The benchmark design is exceptionally fair. Providing identical compute and token budgets across diverse approaches (distillation vs. programmatic heuristics) tests genuine strategic reasoning rather than raw compute hoarding.

Experimental integrity: Thorough evaluation spanning 10 leading models across 60 independent bottling trials. Documenting that 48 out of 60 trials underperformed zero-shot bounds provides a healthy antidote to agent hype.

Writing quality: Cohesive, sharp, and clearly positioned within systems and software engineering literature.

Verdict: strong accept — A groundbreaking benchmark that introduces a crucial missing dimension to agent evaluation: the capacity to amortize intelligence into scalable production code and compact models.

Takeaways

• Stop burning enterprise API credits querying frontier models on millions of repetitive classification turns; prompt your agents to design and distill a local SLM or deterministic script.
• Do not assume that high benchmark reasoning automatically translates to good software engineering; agents require explicit scaffolding to manage budget allocation and training loops.
• Measure your AI systems by their amortized workload cost, not just single-turn accuracy.