DSLE: When the Benchmark Runs at 1x and Nobody Wins

Paper: 2608.09902 Authors: Derin Gezgin, Jim O’Connor, Tanner Goodwin, Gary B. Parker Categories: cs.AI, cs.NE

The Gap

Almost every RL benchmark we love is secretly a benchmark about cheap sampling. The Arcade Learning Environment runs an emulator you can spin at hundreds of times real time and fork into a thousand parallel copies. ProcGen is fast and procedurally generated. VizDoom is fast. MineRL is slower but still an emulated-ish sandbox with a huge human demonstration corpus bolted on. SC2LE is the closest prior art for “real commercial game, real complexity,” but it too can be accelerated and it hands you a structured feature-layer observation plus a fairly informative score signal.

That shared property — you can buy performance with compute — quietly determines what counts as a “solved” benchmark. Sample efficiency becomes a nice-to-have rather than the binding constraint. The gap DSLE goes after is the regime where you cannot buy your way out: a real-time commercial action game, running at 1x, where every environment step is a real button press against a live process, where the reward is essentially a single bit delivered at the end of a one-to-three-minute episode, and where the observation is raw pixels with no privileged feature layers.

There’s a second, smaller gap: reproducibility of these setups. Prior Dark Souls agent work exists mostly as hobbyist repos and one-off scripts tied to a specific machine. Nobody had packaged the thing so two labs could compare numbers.

PROBLEM: our benchmarks are all cheap to sample;
         real-time games with terminal-only rewards are not
   |
   v
ASSUMPTION: a wall-clock-bound, sparse-reward, pixel-only arena
            exposes failure modes that Atari / ProcGen hide
   |
   v
METHOD: containerize Dark Souls: Remastered behind a Gymnasium
        shim -- 22 boss encounters, plus DSLE-5, a 5-boss subset
        chosen for orthogonal difficulty types
   |
   v
EVIDENCE: 5 methods on DSLE-5
          . expert system  -> 63% peak on tutorial boss
          . evolutionary   -> 43% peak on tutorial boss
          . PPO / DQN      -> <= 0.33% tutorial, 0% elsewhere
          . cost           -> tens of wall-clock hours per run
          . all 22 bosses w/ level-50 stats -> a few early wins
   |
   v
CONCLUSION: model-free pixel RL here is not slow, it is *flat*.
            Report survival time + damage dealt, because win rate
            is a constant zero and tells you nothing.

The Increment

One sentence: Before, “hard RL benchmark” meant an environment that needs a lot of samples; after DSLE, there’s a standardized environment where the samples themselves cost real seconds and current model-free methods produce a completely flat learning curve — plus the metrics to describe *how they fail instead of just noting that they do.

Core Mechanism

The engineering core is a container that holds the actual game process plus three bridges to it. An input injector turns discrete actions (attack, roll, block, lock-on, movement) into synthetic keyboard/gamepad events delivered to the running game. A frame grabber pulls the rendered screen as the observation, which is why the agent sees exactly what a human sees — health bars, stamina, boss animations — with no parsed structure. A state reader pulls the small set of scalars needed to compute rewards and terminations: player HP, boss HP, alive/dead flags. Around these sits a thin Gymnasium shim exposing reset / step / observation and action spaces, so anything written against Gym-style APIs plugs in unchanged.

The part that hurts is reset. In an emulator you restore a save state in microseconds. Here you have to bring a commercial game back to a known pre-boss configuration — respawn, walk-in, fog gate, boss re-instantiation — and that is real wall-clock time on every single episode. Combine that with episodes that last tens of seconds and a reward that arrives only at the end, and you get the paper’s central cost fact: tens of hours per training run, which is roughly a rounding error’s worth of experience by Atari standards.

The scientific core is the benchmark design. Rather than dumping 22 tasks and letting everyone cherry-pick, the authors curate DSLE-5 with one boss per failure *type: a plain melee duel, a cramped arena where space is the constraint, a fight where the environment itself kills you, a multi-target fight, and a fast final boss. Then, because every learned method scores zero, they switch the reporting axis to survival time and damage dealt — partial credit that separates “died in eight seconds to a two-enemy pincer” from “stood there for ninety seconds dealing almost no damage.” Those are opposite pathologies and win rate collapses them into the same number.

  +----------------------- container image ----------------------+
  |                                                              |
  |     [ Dark Souls: Remastered process ]   1x real time only   |
  |         ^                          |                         |
  |         | synthetic key / pad      | frames + memory         |
  |         |                          v                         |
  |   [ input injector ]      [ frame grabber ]  [ state reader ] |
  |         ^                          |               |         |
  +---------|--------------------------|---------------|---------+
            |                          v               v
         action                   pixel obs      player HP / boss HP
            ^                          |               / dead flag
            |                          +-------+-------+
            |                                  v
       [ agent ] <--- obs, reward, done --- [ Gymnasium shim ]
                                                  |
                                    reset: respawn + walk-in
                                    (costs real seconds, no rewind)

  reward:  ~0 for the whole episode,  +/- 1 at death or kill
  metrics: win rate  (useless: 0)
           survival time + damage dealt  (the informative pair)

Think of it as a boxing gym that only has one real professional fighter in it. The container is the gym building: standardized ring, standardized lighting, so a trainee’s performance in Cologne means the same thing as in Connecticut. The frame grabber is the trainee’s eyes — they get no telemetry from the opponent’s nervous system, just the same view a spectator has. The input injector is their gloves and feet. The state reader is the scoreboard visible ringside: your HP, their HP, and whether anyone’s on the canvas. The Gymnasium shim is the standard rulebook, so any trainee from any school can step in without renegotiating what a round means.

And reset is the part that makes this gym expensive: you cannot rewind a sparring session. After every knockout, someone has to mop the mat, revive the opponent, and walk the trainee back through the door. The terminal reward is a knockout-only scoring rule — jabs don’t count, only the final result. That’s why the coaches (the authors) add judges’ scorecards: when every trainee loses every bout, “lost” is not a measurement, but “lasted four rounds and landed thirty punches” versus “flattened in eight seconds” is. The DSLE-5 selection is picking five sparring partners who each punish a different weakness — one tests footwork, one tests fighting with your back to the ropes, one tests fighting on a slippery floor, one puts two opponents in the ring, one is simply faster than you.

Key Concepts

• Wall-clock-bound environment: Most RL environments have a knob labeled “go faster.” Emulators run at 1000x; simulators fork into parallel workers; you trade dollars for experience. DSLE has no such knob — the game renders at its own pace and there is no rewind. Concretely: if an episode is 40 seconds of fight plus 20 seconds of reset, you get 60 episodes per hour per instance. PPO on Atari commonly consumes tens of millions of frames; here, tens of hours buys you something like a few thousand episodes. So “PPO didn’t learn” is genuinely ambiguous between “PPO can’t do this” and “PPO never got a real chance” — and that ambiguity *is the property the benchmark is designed to expose, because no amount of engineering removes it.

• Terminal-only sparse reward: Imagine learning chess where the only feedback is “you won” or “you lost” at the end, with no piece values, no captures noted, nothing. Every move in a 60-move game gets equal credit for the outcome. That’s the credit-assignment problem at its worst. Dark Souls bosses are exactly this shape: a two-minute fight is hundreds of decisions and one bit of feedback. Note that DSLE *reads boss HP, so a shaped reward (damage dealt per step) is available — the paper’s default just doesn’t use it, which is both the honest hard version and, as I’ll argue below, a missed baseline.

• Partial-credit metrics for zero-saturated benchmarks: When every method scores 0% you have a measurement problem, not just a performance problem — you can’t rank, can’t detect progress, can’t tell a promising agent from a broken one. The fix is to find intermediate quantities that correlate with progress. Here: how long did you stay alive, and how much damage did you land. Two agents with identical 0% win rates can be worlds apart on those axes, and the failure taxonomy the paper reports (8-second deaths vs. 90-second no-damage stalemates) only becomes visible once you look at them.

Framework Shift

Before (mainstream benchmark):        After (DSLE):

  [ emulator ]                          [ one game, one clock ]
     x1000 parallel copies                 x1  no parallel copies
     speed: 100x-1000x                     speed: 1x, no rewind
     reset: microseconds                   reset: real seconds
                                     
  reward every frame                    reward once, at the end
    +--+--+--+--+--+--+                  ...................+-
    (dense score signal)                 (one bit per episode)

  budget: 200M frames                   budget: hours of wall clock
          (a weekend of GPUs)                   (a weekend, period)

  metric: mean score                    metric: survival time
          (always informative)                  + damage dealt
                                                (because win rate = 0)

  conclusion shape:                     conclusion shape:
    "method A > method B"                 "everything is at zero;
                                           here is *how* each fails"

One sentence: from how many samples does your method need to what can you learn when samples cost seconds and feedback is one bit, the core shift is treating wall-clock cost and reward sparsity as the benchmark’s subject matter rather than as inconveniences to engineer away.

Expert Assessment

Problem choice: The gap is real, though narrower than the framing implies. The genuinely valuable observation is that our field’s difficulty axis has quietly been “sample count” for a decade, and that hides methods’ behavior in the regime where a robot, a lab instrument, or a production system gives you one trial per minute. Dark Souls is a legitimate, well-chosen stand-in for that regime, with the bonus that it comes with a natural difficulty curriculum and a community consensus on what’s hard. The counterargument: you can approximate the same regime by throttling ALE and truncating rewards, and you’d get reproducibility for free. What DSLE adds beyond that is genuine visual and mechanical complexity — plus a licensing headache, since anyone reproducing this needs their own legal copy of the game, which is exactly the kind of friction that killed several prior “real game” benchmarks.

Method maturity: The environment engineering is competent and unglamorous — screen capture, input injection, memory reads, containerization. Standard techniques, well assembled. The *baselines are where I’d push back hard. Reporting that PPO and DQN score 0.33% and 0% is an honest data point, but it’s a weak one, because the obvious ladder of intermediate steps was skipped. The environment already exposes boss HP, so dense damage-based reward shaping is one line away and is the single most likely thing to move the needle; running the flagship result without it makes the negative claim look stronger than the evidence supports. Same for the missing items: no behavior cloning from human play (Dark Souls has effectively unlimited demonstration video), no model-based agent (Dreamer-class methods exist precisely for expensive environments), no temporally abstracted action space, no frame-skip or macro-action study. As it stands, “model-free pixel RL at 1x with terminal rewards and a few thousand episodes fails” is close to a foregone conclusion — nobody in the field would have predicted otherwise. The interesting question is which of the missing rungs is the first to succeed, and the paper doesn’t ask it.

Experimental integrity: Refreshingly honest, and the metric choice is the paper’s best idea. Two caveats. First, statistical strength: at tens of wall-clock hours per run you can’t have many seeds, and a 0.33% win rate is likely a single lucky episode. That number should be reported with a count (“1 win in 300 episodes”), not a percentage that implies precision. Second, the all-22-bosses study at level-50 stats is a nice stress-relief control — it separates “the agent is weak” from “the character is weak” — but advantaged stats also change the fights’ dynamics, so it’s not a clean ablation of policy quality. The comparison between the hand-written expert system (63%) and the evolutionary baseline (43%) is the most informative contrast in the paper, and it’s the one that gets the least analysis.

Writing quality: The abstract is unusually candid, which I appreciate. Where I suspect corners were cut: the environment characterization. Anyone building on DSLE immediately needs numbers the abstract doesn’t hint at — mean step latency and its jitter, reset duration, determinism guarantees, whether the game is paused between steps or runs asynchronously (this determines whether the benchmark is even Markovian in practice), and what happens under frame drops. If I could rewrite one section, it’d be that one; a proper latency-and-determinism appendix is what turns this from a paper into infrastructure other people trust. Second choice: a “what would it take” roadmap section converting the failure taxonomy into concrete hypotheses.

Verdict: weak accept — a useful, well-scoped artifact with commendably honest negative results, held back by a baseline suite that stops one rung short of where the interesting findings begin.

Takeaways

Concrete things worth stealing:

• The three-bridge pattern for wrapping any live application as an RL environment: frame grabber for observations, memory/state reader for reward and termination, synthetic input injector for actions, all inside a container with a Gym shim on top. This transfers directly to GUI agents, legacy desktop software, CAD tools, and any closed-source system with no API. The container is what makes results portable across labs; the state reader is what makes reward computation possible without instrumenting the app.

• Design your metric for the regime you’re actually in. When your headline metric saturates at zero, it has stopped being a measurement. Find intermediate quantities on the causal path to success — here, survival time and damage dealt — and you regain the ability to rank, debug, and detect progress. This is generalizable advice for anyone benchmarking on tasks that are currently out of reach: 0% pass rate across five agents means you need a different y-axis, not a bigger training budget.

• Failure taxonomy as a result. “8-second deaths in cramped multi-target arenas” and “90-second stalemates dealing no damage” are opposite pathologies pointing at different fixes (survival/positioning vs. commitment/aggression). Reporting failure *kinds rather than a single aggregate is a cheap way to make a negative result actionable.

• Curate a small subset with orthogonal difficulty axes. DSLE-5 picks five tasks that each stress a different capability instead of shipping 22 and inviting cherry-picking. This is good benchmark hygiene generally: a small, defensible default suite plus a full set for later, with the selection rationale stated in terms of what each task tests.

• Wall-clock as the budget unit. Reporting “tens of hours per run” instead of “N million steps” is the right currency for any expensive-environment work, and it changes which methods look attractive — model-based and imitation-bootstrapped approaches gain a lot of ground the moment steps stop being free.

What I wouldn’t take: the specific numbers. They tell you that untuned model-free pixel RL with terminal rewards fails here, which you already believed.