EnvACE: Training Agents by Making the Policy Play the Environment

Paper: 2608.06197 Authors: Zishan Xu, Zhiyuan Yao, Yuxin Chen, Yifu Guo, Zhengxi Lu, Yuquan Lu, Jinyang Huang, Yan Xu, Yasheng Wang, Weinan Zhang, Xingshan Zeng, Weiwen Liu Categories: cs.AI

The Gap

Training an LLM agent for long-horizon tool use has a hard prerequisite: something has to answer the tool calls. The field has settled into two ways of providing that something, and both have a ceiling.

The first is to build real or synthesized executable environments. This is what most serious agentic RL work does, and it works — but the cost is brutal and it is the wrong kind of cost. Every new tool domain needs its environment constructed, and then verified, because an environment that returns subtly wrong responses trains an agent to be subtly wrong. Verification does not amortize: it scales with the number of domains you want coverage over. This is why “environment scaling” has become its own research direction, and why the ceiling on agent training increasingly looks like an environment-engineering ceiling rather than a modeling one.

The second is to use an external simulator. Cheaper to run, but hard to ground — the simulator’s notion of how a tool behaves drifts from how it actually behaves, and the policy learns the simulator’s physics instead of the world’s.

Both approaches share a structural assumption worth noticing: that environment dynamics live outside the policy, in a separate artifact that has to be built, maintained, and kept faithful. The knowledge of “what happens when I call this tool with these arguments” sits in the environment, and the policy only ever accesses it through interaction.

Long-horizon tool-use training needs a responder
         |
         +-----------------------+
         |                       |
         v                       v
  executable envs          external simulators
  (costly to build          (cheap but hard
   AND verify)               to ground)
         |                       |
         +-----------+-----------+
                     |
                     v
      Shared assumption: dynamics live OUTSIDE the policy
                     |
                     v
      [This paper] What if the policy holds them?
                     |
                     v
      World rehearsal: policy acts, then plays
      the environment; both roles trained jointly
                     |
                     v
      Evidence: beats environment-scaling baselines on
      BFCL-v4, tau^2-Bench, VitaBench, FinMCP-Bench
                     |
                     v
      An agent world model in the parameters --
      usable at train time AND at test time

The Increment

One sentence: Before this paper, scaling agentic RL meant scaling the environments you train against; after it, the policy can generate its own environment responses during training and carry the resulting world model in its weights.

Core Mechanism

EnvACE replaces external environment interaction during training with world rehearsal. The policy alternates between two roles within a single rollout. First it acts: given the task and history, it generates a tool call. Then it rehearses: it switches roles and plays the environment, producing the response that call would induce. It then conditions its next decision on that rehearsed response, and the loop continues. From the outside, the rollout looks like a normal multi-turn tool-use trajectory; internally, the model wrote both halves of the conversation.

The critical design choice is that both roles are jointly optimized end-to-end using task-success rewards. There is no separate environment-modeling loss, no supervised imitation of real responses. The rehearsal role is trained by the same signal that trains the acting role: did the trajectory succeed at the task. This is what keeps rehearsal honest in the way that matters. A rehearsal that hallucinates a convenient response leads the policy down a path that fails the task, and the task-success reward pushes back on both the action and the fabricated response that enabled it. The environment model is shaped not to be maximally realistic in general, but to be realistic enough that decisions conditioned on it succeed.

The consequence is that dynamics get internalized into the parameters. The policy is not consulting an environment; it holds a model of how actions map to responses. The paper calls this an agent world model, and its most interesting property is that it does not stay confined to training. At test time, the policy can perform private rehearsal before committed execution — imagining what a tool call would return, evaluating whether that leads somewhere useful, and only then actually issuing the call against the real environment. Under a moderate rehearsal budget this yields further gains with no additional external interaction, which is the payoff of having put the dynamics somewhere the model can query for free.

   Training rollout under world rehearsal

   [Task]
     |
     v
  +------------------+
  | ACT role         |  policy emits tool call a_t
  +------------------+
     |
     v
  +------------------+
  | REHEARSE role    |  SAME policy plays environment,
  | (no external env)|  produces response o_t
  +------------------+
     |
     v
  condition on o_t --> next decision a_t+1
     |
     v
   ... repeat to horizon ...
     |
     v
  +----------------------------+
  | task-success reward        |
  | trains BOTH roles jointly  |
  +----------------------------+


   Test time: the internalized model becomes optional lookahead

   candidate call --> private rehearsal --> worth it?
                                             |
                              no <-----------+-----------> yes
                              |                            |
                         revise call              commit to REAL env

Here is the structural metaphor. Think about how a chess player trains without an opponent. The obvious method is to hire sparring partners — real opponents, expensive, and you need a lot of them at many strengths to get coverage. That is the executable-environment approach. The alternative is to buy a chess computer, but a weak or quirky engine teaches you to beat it, not to beat people. That is the external simulator, ungrounded.

What strong players actually do is play both sides themselves. You make your move, then you turn the board around and think as hard as you can about the reply, then you turn it back and respond to that reply. Crucially, you are not trying to produce the reply a specific opponent would make — you are trying to produce a reply good enough that a plan surviving it is a plan worth having. And you keep yourself honest not by grading the realism of your imagined replies, but by whether the resulting lines actually win when you finally play a real game. That is the task-success reward training both roles.

The metaphor also explains the test-time behavior exactly. A player who has trained this way does not stop turning the board around once real games start. Before committing to a move, they play it out in their head first — private rehearsal before committed execution. The board-turning habit built during training became the calculation ability used during play. The same parameters serve both.

Key Concepts

• World rehearsal: The policy generating the environment’s response to its own action, then conditioning on it. The word “rehearsal” is doing precise work here — it is not simulation in the sense of a separate faithful model, and it is not hallucination in the sense of unconstrained invention. It is a dry run, performed by the actor, and judged by whether the performance it prepares for succeeds. A concrete instance: the agent emits a call to a flight-booking API with a malformed date. In a real environment it gets an error response. Under world rehearsal the policy must itself produce that error response — and if it instead rehearses a cheerful success, the trajectory proceeds on a false premise and fails the task, and gradient flows back through the fabricated response.

• Internalized dynamics as an agent world model: The distinction between having access to an environment and having a model of one. An agent with API access knows what happens when it calls, but only by calling. An agent with internalized dynamics knows before it calls. This matters for exactly one reason: knowledge you hold in parameters is free to query, while knowledge that lives behind an interaction is not. Every lookahead step against a real environment costs a call, has latency, may have side effects, and may not be reversible. Every lookahead step against internalized dynamics costs a forward pass. That asymmetry is what makes test-time rehearsal viable at all.

• Private rehearsal before committed execution: The test-time use of the world model, and the reason the training method has value beyond training. “Private” means the rehearsed calls never touch the real environment — nothing is booked, nothing is sent, nothing is deleted. “Committed” means only the surviving call is actually executed. This is a safety-relevant structure that comes along for free: the agent gets to be wrong in imagination rather than wrong in production, which is the single most useful property you can give an agent that operates on real systems.

Framework Shift

Before (environment scaling):        After (world rehearsal):

  policy                              policy
    |                                   |
    | tool call                         | tool call
    v                                   v
  +-------------------+              +-------------------+
  | external env      |              | SAME policy,      |
  | built + verified  |              | rehearse role     |
  | per domain        |              +-------------------+
  +-------------------+                  |
    |                                    | rehearsed response
    | response                           v
    v                                 next decision
  next decision                          |
    |                                    v
    v                              task-success reward
  task-success reward              trains act AND rehearse
  trains policy only                     |
                                         v
  scaling knob:                    scaling knob:
  build more environments          more rehearsal, same weights

                                   bonus: rehearsal survives
                                   into test time as lookahead

From treating environment dynamics as external infrastructure to be constructed, to treating them as knowledge the policy can hold, the core shift is moving the world model from outside the parameters to inside them — where it can be queried for free at inference.

Expert Assessment

Problem choice: The gap is real and the framing is sharp. Environment construction genuinely is the bottleneck in agentic RL right now, and the observation that verification cost scales with domain coverage rather than amortizing is the correct reason it is a bottleneck. Positioning against “environment-scaling baselines” is the honest comparison to pick, since that is the approach with momentum. There is also a satisfying conceptual tidiness to the move: model-based RL has always argued that learning dynamics beats querying them, and this is that argument arriving in the LLM agent setting with the twist that the dynamics model and the policy are the same network.

Method maturity: This is a clever insight rather than brute force, and the cleverness is concentrated in one decision — training the rehearsal role with task-success reward instead of a supervised environment-modeling objective. That choice is what prevents the obvious failure mode where the policy learns to rehearse whatever response makes its life easiest. But I would not call the problem fully solved by it, only fully pressured. Task-success reward penalizes self-serving rehearsals that lead to failure; it does not penalize self-serving rehearsals that lead to success for the wrong reasons. A policy that systematically rehearses a simpler world than the real one, and learns a policy that works in that simpler world, gets rewarded during training and discovers the discrepancy only at deployment. The transferability results across four benchmarks are the relevant evidence against this, and they are reassuring, but the concern is structural rather than empirical and would show up worst exactly where it is hardest to test: novel tools with dynamics unlike anything in pretraining.

Experimental integrity: Four benchmarks spanning general function calling, multi-turn interaction, and a domain-specific financial suite is a reasonable spread, and evaluating against environment-scaling baselines rather than against no-training-at-all is the fair comparison. The controlled studies showing the effect holds across model scales are the most valuable table in the paper for a reader deciding whether to adopt this, because a technique that only works at one scale is usually exploiting something other than what it claims. The result I would probe hardest is the test-time rehearsal gain: “moderate rehearsal budget” is doing unspecified work, and the honest accounting compares that budget against simply spending the same tokens on longer reasoning without the rehearsal framing. Code release helps here.

Writing quality: The abstract is well-constructed and the central mechanism is stated crisply enough to be understood on one reading, which is rarer than it should be. The weakest point is that “internalizes the relationship between actions and their environment responses in its parameters” is asserted more than it is interrogated — the paper would be stronger with a direct probe of what the rehearsal role actually learned, ideally comparing rehearsed responses against real ones on held-out tools. That section would convert the central claim from plausible to demonstrated.

Verdict: weak accept — A genuinely good idea with the right training signal, held back from a stronger verdict by the unexamined question of what happens when the rehearsed world is consistently simpler than the real one.

Takeaways

Three things a practitioner can steal:

1. Train the simulator with the downstream reward, not with a modeling loss. This is the transferable core. Whenever you need a component that stands in for something expensive — an environment, a user, a grader, a retrieval corpus — the instinct is to train it to be accurate. Training it instead on whether the downstream task succeeds under it is a different and often better objective, because it concentrates fidelity where fidelity affects decisions and spends nothing on realism that does not. This applies well outside agents.

2. Separate rehearsal from commitment as a deployment pattern. Even without adopting the training method, the test-time structure is independently useful: let the agent imagine the result of a consequential action, evaluate it, and only then execute against the real system. For any agent touching production — sending messages, moving money, mutating databases — the ability to be wrong in a rehearsal rather than wrong in the world is worth engineering for directly. The paper shows this can be learned rather than hand-built.

3. Notice when your scaling knob is infrastructure rather than compute. The reframe worth internalizing: if progress on your system requires building more artifacts by hand — more environments, more graders, more labeled sets — ask whether the model can hold that knowledge instead. The answer is often no, but the question is asked too rarely, and when it is yes the cost curve changes shape rather than shifting down.