Survey of Optimizers: There Is No Context-Free Replacement for AdamW

Paper: 2608.28557 Authors: Ruoran Xu Categories: cs.AI, cs.LG

The Gap

The received narrative about neural network optimization was that it consisted of a succession of Adam variants — one adaptive moment estimate after another, each claiming a modest improvement. That narrative stopped describing the field, and the survey’s opening move is to say so: optimization in 2025-2026 is no longer well described that way.

What replaced it is a design space that expanded along three axes at once. From coordinates to matrices and layers — the update is no longer elementwise but structured. From fixed training horizons to policies over time — the optimizer’s behaviour is now something scheduled rather than held constant. And from mathematical update rules to state representations that must survive sharding and low-precision computation — the constraints now include whether the optimizer’s own state can be distributed across devices and held in fewer bits.

The gap is that this expanded space had no map. Individual methods were published with claims of improvement, but the field lacked an account of which axes are independent, which methods occupy the same position, and — the practically important part — why the rankings keep changing.

   THE NARRATIVE THAT STOPPED BEING TRUE

   "optimization = a succession of Adam variants"
        -> each claiming a modest improvement
        |     (this stopped describing the field)
        v
   WHAT ACTUALLY REPLACED IT -- three simultaneous expansions

     coordinates  ->  MATRICES AND LAYERS
        the update is no longer elementwise

     fixed training horizons  ->  POLICIES OVER TIME
        the optimizer's behaviour is SCHEDULED

     mathematical update rules  ->  STATE REPRESENTATIONS
        that must survive SHARDING and LOW PRECISION
        (can the state be distributed? held in fewer bits?)

        |
        v
   [GAP] an expanded design space with no map:
         which axes are independent?
         which methods occupy the same position?
         and why do the rankings keep changing?

The Increment

One sentence: Before this survey, the expanded optimizer design space was navigated method by method; after it, four largely independent axes organise the space, and the honest conclusion is that matrix-aware methods are a genuine advance while no context-independent replacement for AdamW exists.

Core Mechanism

The organising contribution is four axes, and their usefulness depends on their being largely independent — a claim that means a method’s position on one axis does not determine its position on another.

• Temporal estimation — how the optimizer estimates the gradient’s relevant statistics over time. This is the axis the Adam-variant era occupied.
• Update geometry — the shape of the update: elementwise, matrix-structured, layer-structured. This is where the matrix methods live.
• Horizon management — how the optimizer’s behaviour changes over the course of training, including schedule-free training.
• Representation and systems — the state representation, subject to sharding and quantisation constraints. This axis exists almost entirely because of distributed and low-precision training.

Situating methods across these axes is what turns a list into a structure. The survey connects the spectral normalization of Muon, the historical matrix statistics of Shampoo and SOAP, adaptive and hybrid matrix methods, memory-efficient optimizers, schedule-free training, small-batch corrections, and quantized optimizer states — each placed relative to the others rather than described in isolation.

Then the conclusion, which is the reason to trust the survey. It is deliberately non-triumphal, and it has two halves.

Matrix-aware methods represent a genuine advance. This matters because a survey that concluded “nothing works” would be as unhelpful as one that concluded “everything works”. The matrix family is identified as real progress, not marketing.

But there is no context-independent replacement for AdamW. The rankings change with model scale, data-to-parameter ratio, batch size, schedule, parameter partition, tuning budget, and whether the target metric is tokens, FLOPs, wall-clock time, or memory. Read that list as the survey’s actual finding: the comparison is not merely noisy, it is multi-dimensional in the objective. Two methods can be correctly ranked in opposite orders by two defensible evaluations that differ only in whether the metric is tokens or wall-clock time.

The practical consequence is a pair of prescriptions. A compositional view of optimizer design — methods as combinations of choices along the four axes rather than as named wholes. And a stricter protocol for evaluating optimizer claims — because if rankings depend on that list of conditions, a paper reporting a single comparison has reported one point in a space.

   FOUR LARGELY INDEPENDENT AXES

   [1] TEMPORAL ESTIMATION
       how gradient statistics are estimated over time
       <- the axis the "Adam-variant" era occupied
   [2] UPDATE GEOMETRY
       elementwise | matrix-structured | layer-structured
       <- where Muon, Shampoo, SOAP, hybrid methods live
   [3] HORIZON MANAGEMENT
       how behaviour changes over training
       <- includes schedule-free training
   [4] REPRESENTATION AND SYSTEMS
       state representation under sharding + quantisation
       <- exists because of distributed / low-precision training

   CONCLUSION (deliberately non-triumphal)
     [a] matrix-aware methods ARE a genuine advance
     [b] but NO context-independent replacement for AdamW
         rankings change with:
           model scale | data-to-parameter ratio | batch size
           schedule | parameter partition | tuning budget
           metric: tokens / FLOPs / wall-clock / memory
         -> the comparison is MULTI-DIMENSIONAL, not just noisy
         -> two defensible evaluations can rank two methods
            in opposite orders

   PRESCRIPTIONS
     - a COMPOSITIONAL view: methods as combinations of
       choices along the axes, not named wholes
     - a STRICTER PROTOCOL for evaluating optimizer claims

Think of it as choosing a vehicle when the question “which is fastest?” has several legitimate answers. Around a track, one car wins; over a rough road, another; on fuel per kilometre, a third; and if you are constrained by what fits in your garage, the winner changes again. A magazine that ranked them in one order would not be wrong so much as under-specified — and a reader who took the ranking as universal would be misled by a correct measurement. The survey’s four axes are the equivalent of describing vehicles by engine, drivetrain, suspension and packaging, so that a buyer can locate their own constraints instead of adopting someone else’s ranking. And the “no replacement for AdamW” conclusion is the observation that the long-standing default is not beaten by anything across the whole space — which is a more useful statement than a leaderboard.

Key Concepts

• Four largely independent axes: temporal estimation, update geometry, horizon management, representation and systems. The independence is the claim that gives the map its value — a choice on one axis does not constrain the others.
• Matrix-aware methods as genuine advance: the positive half of a non-triumphal conclusion. Distinguishing real progress from marketing is what makes the negative half credible.
• Multi-dimensional rankings, not noisy ones: the conditions that flip rankings include the choice of metric — tokens versus FLOPs versus wall-clock versus memory. That is not measurement noise; it is the objective being genuinely multi-objective.
• Optimizer state as a systems constraint: the representation axis exists because state must survive sharding and quantisation. It is the axis that most obviously could not have existed in the Adam-variant era, and it reframes an optimizer as partly a distributed-systems object.

Framework Shift

Before (method by method, single comparison):
  new optimizer published, compared against AdamW
  -> "improvement" reported from one setting
  -> rankings presented as universal
  -> the design space has no map

After (axes plus a stricter protocol):
  4 axes: temporal estimation | update geometry |
          horizon management | representation and systems
  methods placed as COMBINATIONS, not named wholes
  ranking depends on scale, ratio, batch, schedule,
    partition, tuning budget, and WHICH METRIC
  -> evaluate claims across conditions, not at one point

From comparing named optimizers in single settings, to placing design choices on independent axes and reporting which conditions a claim depends on, the core shift is that optimizer performance is conditional, and a comparison without its conditions is under-specified rather than informative.

Expert Assessment

Problem choice: Very good, and it fills a real gap. The optimizer literature had grown faster than its organisation, and the practical question a practitioner faces — “should I switch, and under what conditions?” — could not be answered from individual papers that each report a win.

Method maturity: A survey’s quality is in its axes and its refusal to over-claim, and both are present. Four axes whose independence is asserted and used is a genuinely useful structure, and the non-triumphal conclusion is the harder thing to write: a survey author has every incentive to declare a winner, and declining to do so on the grounds that the ranking is multi-dimensional is the more useful contribution. The compositional view follows naturally from the axes rather than being bolted on.

Experimental integrity: The survey’s evidence is the literature, so its integrity rests on how it characterises that literature. Naming the conditions that flip rankings — including the metric choice — is the key empirical claim, and it is the kind of statement that is falsifiable in principle by finding a method that wins across conditions. The limitation is inherent to surveys: no new experiments, so the conclusion that there is no context-independent replacement is a reading of existing evidence rather than a demonstration, and the list of conditions is as complete as the author’s survey of the field.

Writing quality: The abstract states the conclusion and the conditions in the same breath, which is exactly right for a survey whose value is in the conditional. The four-axis framing scales well to a long document. What would help most is a table placing the named methods on the axes, since that is the artifact a practitioner would actually consult when deciding what to try.

Verdict: strong accept — it gives the expanded optimizer design space a usable structure and resists the temptation to declare a winner where the ranking depends on the objective.

Takeaways

• State the conditions with the ranking. If an optimizer’s position depends on metric, scale or tuning budget, a comparison without those is under-specified.
• Treat the metric as part of the objective, not as a measurement detail. Tokens, FLOPs, wall-clock and memory are different goals, and a method can win on one and lose on another.
• Look for the axes before looking for the winner. Independent design axes let you locate your own constraints instead of adopting someone else’s ordering.
• Do not expect a universal replacement for a long-standing default. When nothing displaces AdamW across the whole space, that is a finding about the space, not a failure to find the right method.