OpenForgeRL: Training Harness-native Agents End-to-End with RL at Scale

Paper: 2607.21557 Authors: Xiao Yu, Baolin Peng, Ruize Xu, Hao Zou, Qianhui Wu, Hao Cheng, Wenlin Yao, Nikhil Singh, Zhou Yu, Jianfeng Gao Categories: cs.AI, cs.CL

The Gap

Modern AI agents live inside elaborate inference harnesses — Claude Code, Codex, OpenClaw — that orchestrate multi-turn reasoning, tool calls, and environment interaction. These harnesses are stateful, multi-process, and deeply entangled with the systems they manage. But here’s the problem: if you want to train an agent with RL inside one of these harnesses, you’re stuck. Existing open RL frameworks (veRL, OpenRLHF, TRL) assume a clean loop — sample from policy, get reward, update parameters. They cannot natively express the messy reality of a harness that spawns sub-processes, makes asynchronous tool calls, and maintains state across dozens of turns. So researchers either (a) train in simplified toy environments and hope it transfers, (b) use proprietary closed systems like OpenAI’s internal stacks, or (c) hack together brittle training pipelines that break when the harness changes. The gap is clear: there’s no open, general-purpose way to do end-to-end RL training inside real-world harnesses at scale.

Problem: Complex harnesses (stateful, multi-process)
  |
  v
Assumption: RL training needs clean, stateless policy loops
  |
  v
Contradiction: Real harnesses don't fit that abstraction
  |
  v
Prior work: Train in simplified envs OR use proprietary infra
  |
  v
Gap: No open framework for harness-native RL training
  |
  v
Method: Lightweight proxy intercepts harness model calls
  + Kubernetes orchestrator isolates rollouts in containers
  |
  v
Evidence: Trains agents in ZeroClaw/OpenClaw/Codex + GUI envs
  + Outperforms open baselines on ClawEval, OSWorld, WebVoyager
  |
  v
Conclusion: Harness-native RL training is now accessible and effective

The Increment

One sentence: Before this paper, training an agent with RL inside a real inference harness required proprietary infrastructure or painful engineering; after this paper, any researcher can do it with an open-source proxy and a Kubernetes cluster.

Core Mechanism

OpenForgeRL’s architecture has two key components that work together to bridge the harness-training gap. The first is a lightweight proxy server that sits between the harness and the language model. When the harness makes an API call to the model (e.g., “generate the next action”), the proxy forwards this to the actual training policy, gets a response, and returns it to the harness — exactly like a normal model server. But simultaneously, the proxy records the full context (system prompt, tool schemas, conversation history, model output) as a training trajectory. This recording is formatted into the standard data structures that RL frameworks like veRL expect. The beauty is that the harness doesn’t know it’s talking to a proxy rather than a real model server — it operates exactly as it would in production.

The second component is a Kubernetes-based orchestrator that manages the rollout process. Each training episode gets its own isolated container: the harness environment runs in one pod, the proxy in another, and they communicate over the network. This isolation is critical because harnesses like browser-use agents or coding agents can crash, hang, or leak state — you don’t want one broken rollout contaminating the entire training job. The orchestrator handles lifecycle management, timeout enforcement, and resource allocation across hundreds of concurrent rollouts. After a rollout completes, the recorded trajectories flow into a standard RL training loop (PPO, GRPO, etc.) hosted by veRL, which updates the policy parameters. Those updated parameters are then served back through the proxy for the next batch of rollouts.

+--------------------------------------------------+
|              Kubernetes Orchestrator              |
|                                                  |
|  +-----------+    +---------+    +-----------+   |
|  |  Harness  |    |  Proxy  |    | RL Trainer|   |
|  |  (e.g.    |--->| (record |--->| (veRL w/  |   |
|  |  OpenClaw)|    |  & fwd) |    |  PPO/GRPO)|   |
|  |  in pod)  |<---|  in pod)|<---|  in pod)  |   |
|  +-----------+    +---------+    +-----------+   |
|       |               |                |         |
|       v               v                v         |
|  [Tool calls]   [Trajectories]   [Policy update] |
|  [Env state]    [to storage]     [to proxy]      |
+--------------------------------------------------+

The structural metaphor: think of OpenForgeRL as a restaurant kitchen with a secret critic. The chef (harness) runs the kitchen exactly as always — same recipes, same workflow, same tools. But there’s a critic (proxy) sitting at the pass, tasting every dish that goes out. The critic doesn’t interfere with the cooking, but meticulously records what was ordered, what was prepared, how it tasted, and whether the diners (environment) were satisfied. After service, the critic sits down with the head chef (RL trainer) and reviews the evening’s performance: “this dish overcooked, that sauce was brilliant, here’s what to adjust.” The next night, the chef’s skills are slightly better. The kitchen (Kubernetes orchestrator) ensures each dinner service is fully isolated — if one table catches fire, the rest of the restaurant keeps running. The key insight is that you don’t need to redesign the kitchen to train a better chef. You just need someone watching carefully and providing feedback.

Key Concepts

• Harness-native training: Traditional RL training simplifies the agent’s operating environment into a clean Markov decision process — state, action, reward, next state. But real agents deployed in harnesses like Claude Code or browser-use frameworks operate in a much messier world: they spawn sub-agents, call external APIs, maintain memory across turns, and interact with tools that have unpredictable behavior. “Harness-native” means training happens *inside the real harness, not a simplified proxy of it. Concretely, if your agent will be deployed using OpenClaw’s tool-calling loop with retry logic and context management, you train it using that exact same loop. The proxy makes this possible by intercepting without interfering.

• Decoupled training and inference: In a standard RL setup, the model that generates actions and the model being updated are tightly coupled — often the same process. OpenForgeRL separates them. The proxy handles inference (generating actions for the harness), while veRL handles training (updating policy parameters from collected trajectories). They communicate asynchronously through shared storage. This decoupling means you can use any harness on one side and any RL framework on the other, and swap either without changing the other. It’s the difference between a monolithic application and a microservice architecture — each component can evolve independently.

• Rollout isolation via containers: Each training episode is a self-contained story with a beginning (task assignment), middle (agent acting in environment), and end (reward computation). In naive implementations, a buggy rollout can corrupt shared memory or leave zombie processes. OpenForgeRL gives each rollout its own container, like giving each patient their own hospital room. If one episode crashes or hangs, the orchestrator kills its container and starts fresh, without affecting the hundreds of other concurrent rollouts. This isn’t just good engineering practice — it’s essential for RL training stability, where a single corrupted trajectory can destabilize the entire policy update.

Framework Shift

Before (mainstream approach):              After (this paper):
+-------------------+                     +-------------------+
| Simplified Env    |                     | Real Harness      |
| (custom Gym-like) |                     | (OpenClaw, Codex, |
|                   |                     |  browser-use, ...)|
+--------+----------+                     +--------+----------+
         |                                         |
         v                                         v
+-------------------+                     +-------------------+
| Custom training   |                     | Proxy (record &   |
| loop, tightly     |                     | forward, transparent|
| coupled to env    |                     | to harness)       |
+--------+----------+                     +--------+----------+
         |                                         |
         v                                         v
+-------------------+                     +-------------------+
| RL update (in     |                     | K8s Orchestrator  |
| same process)     |                     | (isolated pods)   |
+-------------------+                     +--------+----------+
                                                   |
                                                   v
                                          +-------------------+
                                          | veRL / any RL     |
                                          | framework         |
                                          +-------------------+

From training in toy environments and hoping for transfer, to training directly in the production harness — the core shift is treating the harness as a first-class citizen in the RL training loop rather than an obstacle to abstract away.

Expert Assessment

Problem choice: This is a genuine and well-identified gap. The field has been producing increasingly complex agent harnesses while training infrastructure lags behind, stuck in the paradigm of simple environment loops. The pain point is real — anyone who’s tried to do RL training inside a browser-use agent or a coding harness has hit this wall. It sits at the intersection of systems engineering and ML research, which is exactly where practical progress is bottlenecked right now.

Method maturity: The proxy-based decoupling is a clever and principled engineering solution rather than a novel algorithmic contribution. It’s more “infrastructure insight” than “scientific insight.” The Kubernetes orchestration is solid but not novel — it’s applying well-known container orchestration patterns to a new domain. There might be simpler approaches (e.g., a unified API specification that harnesses and RL frameworks both implement), but the proxy approach has the advantage of requiring zero modification to existing harnesses, which is pragmatically valuable.

Experimental integrity: The benchmarks span diverse domains (code execution, GUI interaction, web browsing) which is commendable. The results against open baselines are strong — 37.7 on OSWorld-Verified and 72.3 on WebVoyager are competitive numbers. However, the comparison pool is somewhat thin; they’re mostly comparing against other open-source efforts of similar scale. The comparison to proprietary systems is absent (understandably, since those numbers aren’t public), making it hard to gauge true competitive standing. The ablation on harness choice (showing some harnesses are harder to learn than others) is genuinely useful and adds analytical depth beyond just benchmark numbers.

Writing quality: The paper is well-structured and clearly motivated. The gap between the introduction’s framing and the experimental depth could be tighter — the introduction promises a general framework, but the experiments cover a specific (though diverse) set of harnesses. Section 4 on harness analysis is the most interesting read and could be expanded; it’s where the paper moves from “we built infrastructure” to “we learned something about agents.” The limitation that error recovery remains weak is honestly acknowledged but deserves deeper analysis — is this a training data issue, a reward signal issue, or a fundamental capability ceiling?

Verdict: weak accept — The paper solves a real infrastructure problem with a clean engineering approach and validates it across diverse settings. It’s more of a systems contribution than a scientific one, but the field needs this kind of scaffolding work. The analytical insights on harness difficulty and RL’s effect on agent behavior elevate it beyond pure engineering.

Takeaways

Three concrete ideas worth stealing:

1. The proxy pattern for RL in complex systems: If you’re trying to add RL training to any complex, stateful system (not just LLM harnesses), the “transparent proxy that records while forwarding” pattern is immediately applicable. You don’t need to modify the system — just intercept its model calls.

2. Harness choice as a training variable: The finding that some harnesses are substantially harder to learn from than others is a practical insight. Before investing compute in RL training, evaluate whether your harness’s control flow is learnable — complex branching, opaque error handling, and implicit state make RL harder.

3. Container isolation for rollout robustness: If you’re doing any kind of parallel environment simulation (RL, evaluation, data collection), running each episode in an isolated container with timeout enforcement is a pattern that generalizes far beyond this paper’s specific application. It prevents the class of bugs where one bad episode poisons the batch.