YC-Bench: Testing AI Agents on Year-Long Startup Simulation

Paper: 2604.01212 Authors: Muyu He, Adit Jain, Anand Kumar, Vincent Tu, Soumyadeep Bakshi, Sachin Patro, Nazneen Rajani Categories: cs.CL, cs.AI

The Gap

Existing LLM agent benchmarks test short-horizon tasks: answer a question, complete a coding challenge, book a restaurant. They measure whether agents can execute discrete actions correctly. What they don’t measure is whether agents can maintain strategic coherence when early mistakes compound over hundreds of turns, when feedback is delayed, and when the environment is partially observable. Prior work like WebShop or ALFWorld gives immediate feedback in fully observable settings. SWE-bench tests coding but not resource management under uncertainty. No benchmark asks: can an agent run a business for a year without going bankrupt?

Problem: Short-horizon benchmarks miss compounding errors
   |
   v
Assumption: Long-term planning requires memory + adversarial robustness
   |
   v
Method: Simulate startup over 365 days, 100+ turns, adversarial clients
   |
   v
Evidence: 9/12 models fail, scratchpad usage predicts success
   |
   v
Conclusion: Frontier models lack long-horizon strategic coherence

The Increment

One sentence: Before this paper, we knew LLMs could plan; after it, we know most can’t sustain plans when mistakes accumulate over hundreds of decisions.

Core Mechanism

YC-Bench simulates a startup environment where an agent manages employees, selects task contracts, and maintains cash flow over a simulated year. The environment is partially observable: the agent doesn’t know which clients are adversarial (they pay late or not at all). Each turn represents a day. The agent must hire/fire employees, assign them to contracts, and track finances. Payroll grows with headcount. Poor early decisions (hiring too fast, accepting bad contracts) compound into bankruptcy.

The benchmark measures three capabilities: (1) strategic planning under uncertainty, (2) learning from delayed feedback (a bad contract choice shows consequences weeks later), and (3) memory management across context truncation. Agents can use a scratchpad to persist information when context windows fill up. The environment includes 12 employee types, 50+ contract types, and adversarial clients that test whether agents can detect patterns in payment behavior.

Environment State (Partial Observability)
   |
   v
Agent Decision: [Hire/Fire | Accept Contract | Assign Tasks]
   |
   v
Simulation Step: Update cash, payroll, contract progress
   |
   v
Delayed Feedback: Contract completes -> payment (or not)
   |
   v
Context Truncation -> Scratchpad persistence
   |
   v
Next Turn (repeat 100+ times)

Think of YC-Bench like piloting a ship through a year-long voyage. The agent is the captain. Hiring employees is adding crew members—each one costs daily rations (payroll). Accepting contracts is choosing which cargo to carry—some clients pay on delivery, others are pirates who steal your goods. The ocean is foggy (partial observability): you can’t see which ships are pirate vessels until after you’ve loaded cargo. Early mistakes compound: if you hire too many crew before securing reliable cargo routes, you burn through supplies and sink. The scratchpad is your ship’s log—the only way to remember which clients were pirates when your memory (context window) gets wiped by the fog. Most captains (9/12 models) sink before reaching port.

Key Concepts

• Compounding consequences: In short-horizon tasks, a mistake costs you one turn. In YC-Bench, hiring too aggressively in month 1 means higher payroll every single day for the rest of the year. If you don’t secure enough contracts to cover that payroll, you bleed cash daily until bankruptcy. It’s like taking on a mortgage—the decision compounds. Most benchmarks test “can you solve this puzzle?” YC-Bench tests “can you avoid digging a hole you can’t climb out of?”

• Adversarial client detection: Some clients in YC-Bench pay late or not at all. The agent must learn from delayed feedback: “Client X didn’t pay last time, so I shouldn’t accept their contracts again.” This requires memory (scratchpad) and pattern recognition across dozens of turns. It’s the difference between touching a hot stove once (immediate feedback) and realizing your business partner has been embezzling for months (delayed, pattern-based feedback). 47% of bankruptcies happen because agents keep accepting contracts from clients who never pay.

• Scratchpad as external memory: When context windows fill up (after ~50 turns), the agent loses access to early conversation history. The scratchpad is a persistent text file the agent can read/write to preserve critical information. Think of it like writing notes in a physical notebook versus trying to remember everything in your head. The paper finds scratchpad usage is the strongest predictor of success—agents that don’t write down “Client X is adversarial” or “Current burn rate: $5K/day” fail to maintain strategic coherence when context truncates.

Framework Shift

Before (mainstream approach):        After (this paper):
                                     
Task -> Agent -> Action -> Done      Day 1: Hire -> Payroll +$2K/day
  ^                                    |
  |                                    v
Single-turn feedback                 Day 30: Contract fails -> Cash -$50K
                                       |
                                       v
                                     Day 60: Bankruptcy (compounded error)
                                       ^
                                       |
                                     Delayed feedback loop

From single-turn task completion to multi-turn strategic coherence under compounding consequences, the core shift is testing whether agents can maintain a plan when early mistakes create cascading failures.

Expert Assessment

Problem choice: This is a real gap. Existing benchmarks test “can you do X?” but not “can you sustain doing X when your earlier choices constrain your later options?” The startup simulation is a clever proxy for any domain where decisions compound (infrastructure management, long-term code refactoring, resource allocation). The problem sits at the frontier of agent capabilities—it’s not about scaling up existing benchmarks, it’s about testing a qualitatively different skill.

Method maturity: The design is elegant: adversarial clients and growing payroll create natural pressure without artificial difficulty spikes. The scratchpad mechanism is a clean way to test memory management. However, the paper doesn’t explore whether simpler baselines (e.g., rule-based heuristics like “never hire more than 3 employees” or “blacklist clients after one late payment”) could achieve comparable performance. It’s possible the task is harder than necessary—some failure modes might be artifacts of the simulation rather than fundamental agent limitations.

Experimental integrity: Testing 12 models across 3 seeds is solid. The failure mode analysis (47% bankruptcies from adversarial clients) is the paper’s strongest contribution—it’s not just “models fail,” it’s “models fail in this specific, diagnosable way.” The scratchpad correlation is compelling. One concern: the paper doesn’t report variance across seeds for individual models, so we don’t know if Claude’s $1.27M average is stable or lucky. The baselines are fair (no cherry-picked weak models), but the lack of ablations (e.g., “what if we remove adversarial clients?”) makes it hard to isolate which aspects of long-horizon planning are actually being tested.

Writing quality: The abstract and introduction are crisp. The related work section is thin—it doesn’t engage deeply with prior work on long-horizon planning (e.g., hierarchical RL, options frameworks). The failure mode analysis (Section 5) is excellent and should be expanded. The paper would benefit from a “design principles” section explaining why they chose a startup simulation over other long-horizon domains (e.g., city planning, military strategy). The conclusion is generic—it should propose concrete next steps for improving long-horizon agent capabilities based on the observed failure modes.

Verdict: weak accept — The benchmark addresses a real gap and the failure mode analysis is valuable, but the paper needs stronger baselines and ablations to isolate what’s actually being tested.

Takeaways

Practitioners building LLM agents for long-horizon tasks should steal the scratchpad pattern: give agents a persistent external memory they can read/write to across context truncations. The paper shows this is the strongest predictor of success. Second, the adversarial client detection failure mode suggests agents need explicit mechanisms for pattern recognition across delayed feedback—don’t assume the model will “just learn” from history. Consider adding a structured memory layer (e.g., a database of past interactions) rather than relying on raw context. Third, the compounding consequences design is a template for creating realistic benchmarks in other domains: identify decisions where early mistakes constrain later options (infrastructure provisioning, technical debt, hiring pipelines) and test whether agents can avoid digging holes they can’t escape.