Reinforcement Learning Finds Stability Solutions Beyond Physics: Using Lyapunov Exponents as Universal Rewards

Paper: 2607.14001 Authors: Slava Andrejev Categories: cs.LG

The Gap

Existing reinforcement learning for physical stabilization has a dirty secret: reward engineering is a black art.

The classic approach to stabilizing an inverted pendulum through rapid vertical oscillation of the pivot point — the Kapitza pendulum — dates back to the 1950s. It works through parametric excitation: by vibrating the support faster than a critical frequency, the inverted position becomes stable. Beautiful physics. But the resulting motion is inherently oscillatory — the pendulum never achieves true static equilibrium pointing straight up.

RL approaches to stabilization typically rely on carefully crafted reward functions: penalize deviation from vertical, penalize angular velocity, add energy costs. These are task-specific, require manual tuning, and provide sparse feedback early in training. Worse, they embed our assumptions about what the solution should look like, potentially constraining the agent to rediscover known strategies rather than finding better ones.

The gap is clear: we lack a universal, physics-informed reward signal that measures stability directly without prescribing what the stable solution looks like.

Problem:  Inverted pendulum stabilization
          |
          v
Assumption: Stability = negative Lyapunov exponent
          |        (universally true for dynamical systems)
          v
Method:   Use LCE as dense RL reward
          |        (no task-specific engineering)
          v
Evidence: Agent finds Kapitza oscillation
          |   AND discovers static upright stabilization
          v
Conclusion: LCE is universal physics-informed reward
            that lets RL exceed human-designed solutions

The Increment

One sentence: Before this paper, physics-informed RL rewards were either task-specific or sparse; after, we have a universal stability metric (LCE) that serves as dense reward and lets RL discover solutions beyond known physics.

Core Mechanism

The Lyapunov Characteristic Exponent measures how fast two initially-close trajectories diverge (or converge) over time. A negative LCE means perturbations decay — the system is stable. A positive LCE means chaos. This is not an engineering heuristic; it’s a fundamental property of dynamical systems.

The core idea is brutally simple: instead of telling the RL agent “stay upright” with a handcrafted reward, tell it “make the Lyapunov exponent as negative as possible.” The agent receives dense feedback at every timestep because LCE can be estimated incrementally. As the agent’s control policy improves, nearby trajectories converge faster — and the reward signal reflects this immediately.

The RL agent (presumably a standard policy gradient or actor-critic method) explores the action space of vertical oscillation parameters. Each episode, the system computes the LCE for the controlled pendulum dynamics. More negative LCE → higher reward. The agent has no knowledge of the Kapitza pendulum solution; it simply optimizes for maximum trajectory convergence.

LCE-based RL loop:

  [Agent Policy] --action--> [Pendulum + Vertical Drive]
        ^                            |
        |                            v
  [LCE Calculator] <--trajectory--- [State Trajectory]
        |
        v
  [Dense Reward = -LCE]
        |
        +------> [Policy Update]

Here’s a metaphor that makes this click: imagine you’re training a tightrope walker, but instead of scoring them on “stayed on the rope” (binary, sparse), you give them a balance meter that shows how quickly they’d recover if nudged at any moment. Every second, the meter updates. The walker doesn’t need to know why a particular strategy works — they just learn to keep that meter reading as high as possible. The Lyapunov exponent is that balance meter for dynamical systems. It doesn’t care how you stabilize; it just measures how stable you are. So the RL agent discovers strategies that no human tightrope coach would have taught — not because humans are wrong, but because the meter reveals stability patterns we’d never think to look for.

Key Concepts

• Lyapunov Characteristic Exponent (LCE): Take two nearly-identical starting states — say, a pendulum at 179.9° and 180.1° from vertical. Run them forward with the same control inputs. If the angle gap grows over time, the system is unstable (positive LCE). If it shrinks, it’s stable (negative LCE). The LCE quantifies this growth/shrink rate *per unit time. Think of it like compound interest: small differences either explode or decay exponentially. A healthy bank account (stable system) has negative “interest” on perturbations — disturbances die out. An unstable system amplifies every tiny wobble.

• Kapitza Pendulum: A pendulum with its pivot point oscillating rapidly up and down. Counter-intuitively, this can make the inverted (upside-down) position stable — like balancing a broom on your palm while bouncing your hand. The physics: rapid oscillation creates an “effective potential” that acts like a bowl, with the inverted position at the bottom. But the pendulum still oscillates *around vertical; it doesn’t sit perfectly still. This paper’s agent found something beyond this classic solution.

• Dense Reward in RL: Most RL problems give feedback only at episode end (win/lose). Dense rewards give feedback every timestep — like a coach yelling tips during practice instead of only at the game’s end. Dense rewards make learning dramatically faster because the agent knows immediately whether each action helped. The challenge: designing dense rewards that actually guide toward the goal. That’s where LCE shines — it’s inherently dense and inherently measures what we want (stability).

Framework Shift

Before (mainstream approach):        After (this paper):

  [Hand-designed reward]              [LCE = universal physics metric]
       |                                    |
       v                                    v
  Task-specific:                     Measures stability directly:
  - "stay near vertical"             - "perturbations decay"
  - "minimize velocity"              - Dense signal, no assumptions
  - "limit energy"                   |
       |                             v
       v                             RL discovers solutions
  RL rediscovers known               we didn't know existed
  physics (Kapitza oscillation)       (static upright stabilization)
       |
       v
  Ceiling: human knowledge

From task-specific reward engineering to physics-informed universal metrics, the core shift is letting the math define what “good” means instead of letting human assumptions constrain the search.

Expert Assessment

Problem choice: Real gap, but narrow. The inverted pendulum is the “Hello World” of control theory, and the physics community has studied the Kapitza pendulum for 70+ years. Using LCE as a reward is a genuinely useful idea, but the paper demonstrates it on only one system. The trajectory matters: if this generalizes to higher-dimensional systems (robotic arms, locomotion), it’s a significant contribution. On a single pendulum, it’s a nice proof of concept.

Method maturity: Clever insight, straightforward execution. The idea of using LCE as a reward is elegant — it’s one of those “why didn’t anyone do this before?” ideas. The RL part is likely standard (the paper doesn’t appear to introduce novel algorithms). There may be simpler approaches: classical Lyapunov function construction methods already exist, though they require mathematical expertise that RL doesn’t. The tradeoff is legitimate — LCE is computable automatically while Lyapunov functions require human derivation.

Experimental integrity: This is where I have questions. The abstract claims the agent “damped the pendulum’s pivoting, leaving it in a strictly upright position.” This needs careful scrutiny: is this truly static equilibrium, or is the oscillation amplitude just very small? The distinction matters enormously. Also, single-author preprint without apparent code — reproducibility is unverified. I’d want to see: (1) phase portraits of the discovered solution vs. Kapitza, (2) robustness to perturbations, (3) comparison against well-tuned conventional RL rewards.

Writing quality: Can’t fully judge from abstract alone, but the framing is clean. If I were rewriting this, I’d add a section explicitly connecting to classical control theory — Lyapunov’s direct method is well-established, and the paper should position itself as “we automated Lyapunov analysis via RL” rather than inventing something disconnected from the literature.

Verdict: weak accept — The core idea (LCE as reward) is valuable and potentially generalizable, but the experimental scope is too narrow to be compelling yet. Needs at least one more dynamical system and rigorous comparison with tuned conventional rewards.

Takeaways

1. Steal this framing: “Universal physics metrics as RL rewards” is a transferable pattern. Any domain where you can compute a stability/chaos metric (fluid dynamics, network routing, power grids) could use this approach.

2. LCE is computable and dense: If you’re working on control tasks where stability matters, LCE estimation is cheap and gives per-timestep feedback. Consider replacing hand-designed reward shaping with LCE (or related metrics like spectral radius).

3. RL can exceed textbook physics: The most provocative takeaway: by not telling the agent *what the solution looks like, it found something better than the known solution. This suggests we should use RL as a discovery tool, not just an optimization tool — let it explore the space of possible solutions without our assumptions about the shape of the answer.