Lighthouse RL: Strategic Resets for Sample-Efficient Circuit Design

Paper: 2607.14008 Authors: Mustafa Emre Gürsoy, Stefan Uhlich, Ryoga Matsuo, Yağız Gençer, Arun Venkitaraman, Chia-Yu Hsieh, Andrea Bonetti, Eisaku Ohbuchi, Lorenzo Servadei Categories: cs.LG, cs.AR

The Gap

Existing methods for analog circuit sizing face a fundamental trade-off. Traditional optimization techniques, like Bayesian Optimization, are sample-efficient for a single target but struggle to generalize when design specifications change. On the other hand, standard Reinforcement Learning (RL) can learn generalizable policies but is notoriously sample-inefficient—it wastes vast amounts of computational effort exploring unpromising regions of the vast design space from scratch. The paper identifies this specific boundary: the inability of current RL approaches to strategically reuse past successes to guide future exploration, leading to poor sample efficiency in expensive black-box optimization.

Problem: High-cost, black-box analog circuit optimization.
    |
    v
Assumption: Standard RL explores randomly from a fixed start.
    |
    v
Method: Introduce "Lighthouse" states (best-found designs) as strategic reset points for new episodes.
    |
    v
Evidence: Tests on benchmark & real circuits show faster convergence, higher success rates, better generalization vs. standard RL and Bayesian methods.
    |
    v
Conclusion: A strategic, memory-augmented reset mechanism can bridge the sample-efficiency gap in RL-based design optimization.

The Increment

One sentence: Before this paper, RL for circuit design was a data-hungry brute-force search; after, it becomes a guided, memory-efficient exploration that builds upon its own best discoveries.

Core Mechanism

The core of Lighthouse RL is its strategic reset strategy. In standard RL, each training episode typically starts from a fixed, often random, initial state. This is like sending a scout into a vast forest from the same starting point every time, regardless of where they found berries yesterday. Lighthouse RL changes this by maintaining a memory bank of high-performing states encountered during training—the “lighthouses.”

During training, after a certain number of episodes, the algorithm selects one of these lighthouse states. It then resets the environment to this state to begin the next episode. The RL agent now starts its exploration much closer to a good solution, allowing it to refine and explore the promising neighborhood in greater detail. The data flow is a cycle: the agent explores, discovers good states (which become lighthouses), and uses those states as new launch pads for future exploration, progressively climbing the performance landscape.

[RL Agent] --> (Action) --> [Environment]
     ^                           |
     |                           v
     +--- (Reward) <------------+
     |
     |
     v
[Lighthouse Memory] <-- (High-Performance State)
     |
     | (Reset)
     v
[Start New Episode from Lighthouse State]

Think of it like a team of architects designing a skyscraper. The standard RL approach is to give them a blank sheet of paper for every new draft, starting from the foundation each time. The Lighthouse approach is to keep their best draft on the table. When they start a new draft, they don’t start from scratch; they start by copying their current best design and then making tweaks and bold modifications from there. The “lighthouse” is their current best blueprint—it doesn’t limit their creativity but ensures they’re always building upon their most advanced work, not rediscovering basic principles. This is how they explore the design space more efficiently.

Key Concepts

• Strategic Reset: In RL, a “reset” is putting the environment back to a starting state. A *strategic reset means doing this intelligently, not randomly. Imagine you’re looking for the highest peak in a foggy mountain range. A random reset means being airdropped to a random location after each attempt. A strategic reset means being dropped back to the highest point you’ve previously reached. You use your own progress as a launchpad, which is a simple but powerful idea for focusing search efforts.
• Lighthouse State: This is just a fancy name for “a really good configuration we found earlier.” They serve as beacons in the vast, dark design space. The key insight is that a good solution is not just an end goal; it’s a valuable *starting point for finding an even better solution. It transforms the optimization from a memoryless search into a cumulative, iterative process of refinement.

Framework Shift

Before (mainstream approach):        After (this paper):
                                     
Episode Start State: Fixed/Random    Episode Start State: Best-Found ("Lighthouse")
        |                                      |
        v                                      v
[Exploration] --> [Terminal/Reset]   [Exploration] --> [Update Lighthouse Memory]
(From same base)                     (From advanced base)
                                     
Focus: Exploit general policy.       Focus: Exploit policy AND memory of good states.

From memoryless, random-start exploration to memory-augmented, strategic-start exploration, the core shift is treating the search history itself as a navigational asset.

Expert Assessment

Problem choice: This is a real and practical gap. Analog circuit design is a canonical expensive black-box optimization problem. Making RL more sample-efficient here has direct industrial value. The problem sits at a crucial intersection of ML efficiency and EDA tool development.

Method maturity: Clever insight over brute force. The idea of resetting to good states is elegant and conceptually simple, which is a strength. It’s more of a strategic wrapper around existing RL algorithms than a fundamental new algorithm. One might ask if simpler warm-starting or curriculum learning approaches were thoroughly explored, but the “lighthouse” framing is clean and effective.

Experimental integrity: The baselines are fair—comparing against standard RL (PPO) and Bayesian Optimization is appropriate. The metrics (success rate, sample efficiency, generalization) are relevant. The improvements are significant (e.g., 100% vs 0-87% success rate). A potential red flag: the benchmarks include a 2D toy problem, which is good for clarity but easy to overfit a clever heuristic to. The real test is on the analog circuits, and there the results look compelling.

Writing quality: The paper is well-structured and clear. The method is explained intuitively. Where it could be stronger is in the *Discussion section. It doesn’t deeply analyze why certain lighthouses are more useful than others, or how sensitive the method is to the lighthouse selection strategy. Rewriting that section to explore failure modes and design principles for the memory bank would elevate it from a good methods paper to a significant contribution.

Verdict: Weak accept — It presents a clean, practical, and demonstrably effective trick that addresses a real bottleneck. The idea is transferable and the experiments are solid, even if the theoretical novelty is incremental.

Takeaways

The most valuable steal from this paper is the general principle of using curated historical successes as strategic starting points for exploration. This isn’t just for RL or circuits. Practitioners working on any expensive optimization problem (e.g., hyperparameter tuning for large models, material discovery, robotics control) can implement a “lighthouse memory” to guide their search algorithms. The specific technique of resetting episodes to past good states is a plug-and-play enhancement one can add to any RL-based optimizer to potentially gain significant sample efficiency.