RevengeBench: Reverse Engineering Code-Space Policies from Behavioral Experiments

Paper: 2606.26094 Authors: Babak Rahmani, Sebastian Dziadzio, Joschka Strüber, Sergio Hernández-Gutiérrez, Matthias Bethge Categories: cs.LG

The Gap

Existing work in inverse reinforcement learning, behavior cloning, and imitation learning assumes either full access to expert trajectories or a known reward function. When the underlying policy is a hidden program (e.g., an LLM-generated decision rule), these methods collapse because the mapping from observations to internal logic is not smooth or parametric. Moreover, prior benchmarks for policy reconstruction do not allow the learner to intervene — they treat observation as passive. This paper fills that gap by formalizing the inverse problem in code-space with a benchmark that explicitly rewards active experiment design: the learner can craft custom opponents to probe the hidden policy.

[Problem: Hidden programmatic policy unknown]
     |
     v
[Assumption: Only behavioral traces vs. sampled opponents are visible]
     |
     v
[Method: Learner designs custom opponents as experiments, observes traces, then submits executable hypothesis]
     |
     v
[Evidence: Continuous action-distance metrics + downstream tournament win-rate]
     |
     v
[Conclusion: Reverse-engineering code-space policies is tractable and benefits from intervention]

The Increment

One sentence: Before this paper, inferring hidden mechanisms from behavior was limited to passive observation or parametric models; after, we can actively design experiments to reconstruct exact executable code policies.

Core Mechanism

The method is structured as a three-phase benchmark. First, an observation phase: the hidden target policy plays against a fixed set of stock opponents, generating behavioral traces (state-action sequences). Second, an experiment design phase: the learner (e.g., an LLM) is allowed to propose custom opponent policies that the target will play against. These opponents are not arbitrary — they must be submitted as executable code, so the learner can carefully design probing strategies (e.g., always move left, or use a specific opening). Third, a reconstruction phase: the learner submits a hypothesis — a piece of code that claims to reproduce the target’s behavior. The quality is measured by continuous action-distance between the hypothesis and the true target over a held-out set of test games.

[Observation Phase]
Target policy ---> vs. stock opponents ---> traces
     |
[Experiment Design Phase]
Learner --> custom opponents --> vs. target --> new traces
     |
[Reconstruction Phase]
Learner --> hypothesis code --> compare with target via distance metric
     |
[Validation] Downstream tournaments: hypothesis vs. other policies

Structural metaphor: Think of the target policy as a secret chess opening book — a sequence of rules dictating which move to play in each position. You, the learner, are a grandmaster who wants to reconstruct that book. You cannot peek inside; you can only play games against it. First, you play a few random openings (stock opponents) and record the secret book’s replies. Then, you deliberately choose unusual moves (custom opponents) to see how the book reacts in rare positions. After collecting enough “behavioral probes,” you write down what you think the rules are (hypothesis code). Finally, you submit that book and see how accurately it predicts the secret book’s moves in new, unseen positions. If your reconstructed book wins against other books in a tournament, that’s extra validation. The mapping: target policy → secret book, stock opponents → random opening games, custom opponents → probing moves, hypothesis code → your reconstructed book, distance metric → prediction error, tournament → your book vs. others.

Key Concepts

• Behavioral Experiments: Unlike passive observation, here the learner can *design the conditions under which they observe the target. This is the reverse-engineering analogue of perturbation experiments in biology. For example, you can force the target to face an opponent that always moves left, and see how the target responds — this disambiguates whether the target has a “dodge left” rule or a “dodge right” rule. Without such an intervention, both rules might produce similar behavior against random opponents.

• Code-Space Policy: The target’s decision rule is represented as *executable code (e.g., a Python function), not as a neural network or a set of parameters. This means the reconstruction problem is discrete (syntax) and compositional. A small change in code can produce a large change in behavior. This differs from standard imitation learning where the policy is a continuous function approximator.

• Elo-Calibrated Policies: The benchmark generates policies using LLMs and then adjusts their strength via Elo ratings so that they are all competitive but not trivial to beat. This ensures that the reconstruction task is neither too easy (because the policy is too predictable) nor too hard (because it’s superhuman). It also prevents learners from just copying a winning strategy — they have to recover the *specific policy.

Framework Shift

Before (mainstream inverse RL):
[Observed trajectories] ---> [Parametric model (e.g., neural policy)] ---> [Behavior prediction]
(No ability to intervene; model is a black-box function)

After (this paper):
[Observed trajectories] + [Designed experiments] ---> [Executable code hypothesis] ---> [Behavior prediction + interpretable rules]
(Intervention actively disambiguates; output is transparent code)

From passive observation plus parametric inference to active intervention plus programmatic reconstruction — the core shift is that the learner is no longer a passive statistician but an experimentalist who can design probes to isolate underlying rules.

Expert Assessment

Problem choice: Real and well-motivated gap. The inverse problem of recovering programmatic policies from behavior is relevant to interpretability, opponent modeling, and AI safety. The paper sits at a natural intersection of program synthesis and behavioral cloning, and the community needs benchmarks like this.

Method maturity: Clever benchmark design, but the paper does not propose a new reconstruction algorithm — it evaluates existing LLMs. The benchmark itself is the contribution. That’s fine for a benchmark paper, but the lack of a dedicated reconstruction baseline (e.g., program synthesis with search) is a missed opportunity. The brute-force approach of asking LLMs is interesting but not optimized.

Experimental integrity: Baselines are fair — they compare numerous LLMs on the same task. The use of continuous action-distance and tournament validation is solid. One red flag: the benchmark only includes 75 policies across 5 environments. Generalizability to real-world game AIs or other domains is unknown. Also, the evaluation code is not yet public (as of the preprint).

Writing quality: Clear structure, but the “Core Mechanism” section could be more explicit about the exact prompt format used for LLMs. The metaphor is helpful, but the paper itself doesn’t include it — you’d have to derive it. Rewriting the experimental design section with a concrete example would elevate it.

Verdict: weak accept — a useful benchmark that opens a new direction, but limited in scope and without a novel algorithmic contribution.

Takeaways

• Practitioners can borrow the idea of experiment design as prompt engineering: if you want to reverse-engineer a black-box agent, craft queries (opponents) that force it to reveal its decision boundaries. This transfers to any domain where you can interact with the target.
• The continuous action-distance metric is a good template for evaluating programmatic reconstructions: instead of exact match, measure how close two policies behave over a test set. This is more robust to trivial differences.
• The Elo-calibration preprocessing ensures that the benchmarking task is discriminative — a technique worth adopting when building any policy-recovery benchmark to avoid ceiling/floor effects.
• If you’re building an AI assistant that needs to infer user preferences from interaction, the active probing loop here is directly applicable: the assistant should ask targeted questions, not just passively observe.