Agency Transfer: How to Upgrade a Suboptimal Policy Without Starting from Scratch

Paper: 2606.09825 Authors: Anton Bolychev, Georgiy Malaniya, Sinan Ibrahim, Pavel Osinenko Categories: cs.LG, cs.AI, eess.SY, math.OC

The Gap

Reinforcement learning from scratch is expensive – you need careful reward shaping, environment tweaking, and enormous compute. But many real-world control tasks already have a working (if suboptimal) baseline policy: a hand-coded controller, a rule-based system, or a previously trained agent. Existing methods either ignore the baseline entirely (start from scratch) or use it only for initialization / imitation learning, which wastes its critical property – it can reliably reach the goal and stay there.

Prior approaches like PPO or SAC start with random exploration, which often fails to reach the goal in early training. Imitation learning (BC) can copy the baseline but then struggles to improve beyond it. No existing model-free method explicitly exploits the fact that the baseline is functional (goal-reaching) to ensure high success rates during the entire training process.

This paper plugs that hole.

[Problem]                        [Assumption]
   |                                 |
 Training from scratch is              A "functional" baseline
 costly and sample-inefficient         policy already exists that
 during early learning.                reaches goal + stays with
                                       high probability.
   |                                 |
   +--- (gap: how to exploit ------>+
        baseline's goal-reaching
        property during training?)
         |
         v
    [Method - Agency Transfer]
      - Two policies: baseline (fixed) + learning (trainable)
      - Arbitrator decides which action to use at each step
      - Probability of using baseline starts high, decays to zero
      - Learning policy trained off-policy from all experience
         |
         +--- [Evidence] ------------+
              High goal-reaching      |
              rate from first         |
              episode. Final policy   |
              outperforms baseline.   |
         v
    [Conclusion]
 Efficient, safe-exploring policy enhancement
 for settings where a functional baseline exists.

The Increment

One sentence: Before this paper, any model-free RL method that wanted to improve a baseline had to either discard the baseline’s goal-reaching guarantee during training or only imitate it; after this paper, you can seamlessly transfer agency from baseline to learner while keeping the goal reached nearly all the time.

Core Mechanism

The method consists of three components: a baseline policy π_b (fixed, suboptimal but functional), a learning policy π_θ (a neural network trainable via any off-policy RL algorithm), and an arbitrator that probabilistically selects which policy’s action is executed at each timestep. The arbitrator’s probability of using the baseline follows a simple schedule:

p_t(baseline) = max(0, 1 - 2 * (t / T)) (linear decay from 1 to 0 over the first half of training), after which the baseline is never used.

Why linear? Because the baseline ensures the agent rarely leaves the goal set – even when the learning policy’s early actions are terrible, the arbitrator often falls back to the safe baseline action. As training progresses, the learning policy accumulates experience (including successful trajectories) and improves. By the time the baseline is fully withdrawn, the learning policy has learned to reach and stay in the goal set on its own.

The learning policy is updated using a standard off-policy algorithm (e.g., SAC) on the replay buffer containing all past transitions, regardless of which policy generated them. The paper provides theoretical analysis showing that if the baseline is functional (i.e., goal-reaching probability ≥ some δ), then the arbitration ensures a high goal-reaching rate during training, and under assumptions the final standalone policy inherits a lower bound on goal-reaching probability.

          +------------------+
          |  Environment     |
          +--+-------+-------+
             ^       |
             |       v
  +----------+--+   +--+-----------+
  | Baseline π_b |   | Learning π_θ |
  +-------+------+   +------+------+
          |                 |
          +--------+--------+
                   |
          +--------v--------+
          |   Arbitrator    |
          | (probabilistic  |
          |  switch based   |
          |  on schedule)   |
          +--------+--------+
                   |
                   v
          +------------------+
          | Selected Action  |  -> to Environment
          +------------------+
          |
          +--- (s,a,r,s') -> Replay Buffer -> RL Update of π_θ

Now let’s ground this with a structural metaphor: think of a car with two drivers – an experienced co-pilot (baseline) and a student driver (learning policy). The co-pilot is cautious and can always bring the car back onto the road (reach the goal). The student wants to learn to drive efficiently. There’s a switch (arbitrator) that decides who takes the wheel at each moment.

At the start, the co-pilot takes over most of the time – every time the student would steer into a ditch, the switch lets the co-pilot correct it. The student still “drives” occasionally, building muscle memory (collecting experience). Over time, the student improves, and the switch gradually lets the student drive more. By the end, the co-pilot is removed entirely, and the student drives solo, having learned from both their own mistakes and the co-pilot’s example.

The magic is that the co-pilot’s guarantee (always able to reach the goal) is preserved throughout training because the switch kicks in whenever the student’s action would be catastrophic. The paper formalizes “functional” as: under π_b, the agent reaches the goal set and stays there with probability ≥ 1 - ε within a bounded time.

Key Concepts

• Functional Baseline: Not just any old policy – a baseline is “functional” if, when executed from any state in the goal set, it keeps the agent there, and from any other state, it reaches the goal set within a bounded horizon with high probability. This is the crucial assumption. Example: a hand-coded hover controller that can stabilize a drone near a target, but flies too slowly and uses too much battery. It works – it just isn’t optimal.

• Agency Transfer: The act of gradually shifting control from baseline to learner. The paper implements it via a time-dependent switching probability. This is distinct from resetting to the baseline or using baseline actions as targets – the learner gets to “feel” the consequences of its own actions, but the safety net of the baseline is always there until withdrawn.

• Goal-reaching Rate: A metric that measures the fraction of episodes where the agent reaches the goal set and stays there. The paper shows that the proposed method maintains this rate near 100% throughout training, while standard RL methods (which explore suboptimal actions early) often dip to 0% in early episodes. This is critical for real-world applications where we cannot afford a crash.

Framework Shift

Before this paper, the mainstream approach was either: a) train from random initialization (hard exploration), or b) imitate the baseline with behavior cloning then fine-tune (slow improvement, no goal guarantee). This paper pivots from “learn by exploring from scratch” to “learn by safely expanding on an existing guarantee.”

Before (mainstream):
   [Start: random policy] ---> [exploration: mostly fails to reach goal]
        |                        |
        v                        v
   [Many episodes of failure] --> [eventually learn goal reaching]
        |
        v
   [Final policy: maybe better, but training was dangerous]

After (this paper):
   [Start: baseline in control] ---> [baseline ensures goal is reached]
        |                                |
        v                                v
   [Learning policy gradually takes over] +--> [goal still reached]
        |
        v
   [Final policy: standalone, better than baseline, training was safe]

One sentence: From learning from scratch with costly exploration to leveraging a functional baseline as a safety net and gradually transferring control, the core shift is agency transfer – treating the baseline not as a teacher to imitate, but as a guarantor of success during training.

Expert Assessment

Problem choice: Real and well-defined. Many control systems come with an existing controller that works but is suboptimal (e.g., in robotics, autonomous driving, process control). Ignoring that baseline is wasteful. The paper correctly identifies the gap: using the baseline’s functional property is more powerful than using it just as initialization or demonstration.

Method maturity: Clever insight, simple implementation. The arbitration schedule is almost too simple – linear decay to zero. Could there be adaptive schedules that work better? Probably yes, but the authors chose simplicity to keep the theoretical analysis tractable. No brute-force tuning; the core idea is elegant. However, the method is essentially “use baseline as an oracle during training” – I’m surprised no one published this exact trick before.

Experimental integrity: Baselines include SAC, PPO, and a version of SAC with pretraining (fine-tuning from baseline). The numbers show the proposed method (called “Agency Transfer SAC”) matches or exceeds returns while achieving far higher goal-reaching rates during training. No red flags, but the paper uses a specific continuous-control domain (robotic reaching tasks with goal sets). I’d like to see more diverse benchmarks (e.g., tasks without a clear goal set, or with stochastic dynamics). The code is not provided in the abstract, but assuming it’s available – worth checking.

Writing quality: The paper is clearly structured – problem statement, formalization, method, theory, experiments. The theoretical section is dense but the core logic is accessible. The main weakness is that the arbitration mechanism’s rationale is explained qualitatively but not justified empirically against alternatives (e.g., constant p, adaptive p). I’d rewrite the “Why Linear?” paragraph to include ablation. Also, the paper claims “high goal-reaching rates” but doesn’t define the threshold – 90%? 99%? In experiments, it’s effectively 100% for the proposed method, but the theory only gives lower bounds.

Verdict: weak accept – a solid incremental contribution with practical value, clean theory, and convincing experiments. Not a paradigm shift, but a useful tool for practitioners who already have a working baseline.

Takeaways

• Steal the agency-transfer schedule: If you have any “safe” fallback policy (even a simple rule), you can use it during training to avoid catastrophic failures. The linear decay is a good default, but you can adapt it based on learning progress. This is directly transferable to any RL problem with a safety constraint.

• Measure goal-reaching rate during training: Most papers report only final returns. This paper shows that intermediate goal-reaching rates are a strong indicator of practical usability (especially in real-world deployment). You can adopt this metric for your own work.

• Theoretical guarantee trick: The paper proves that if the baseline is functional, the learning policy inherits a lower bound on goal-reaching probability after training. This style of analysis – leveraging a “good enough” pretrained component to bound the learner’s performance – can be applied in other RL settings (e.g., reward shaping, transfer learning).

Consider using this method whenever you have a legacy controller that works but you want to improve it with deep RL without risking catastrophic failures during training.