Muon Over AdamW: A Systematic Benchmark of Optimizers for Tabular Deep Learning

Paper: 2604.15297 Authors: Yury Gorishniy, Ivan Rubachev, Dmitrii Feoktistov, Artem Babenko Categories: cs.LG

The Gap

Tabular deep learning has seen extensive architecture innovation—ResNet-like skip connections, attention mechanisms, specialized normalization layers. Yet everyone just uses AdamW to train these models. It’s the inherited default from computer vision and NLP, never questioned in the tabular context. Meanwhile, new optimizers like Muon, Shampoo, and others have shown promise in vision and language domains, but nobody’s checked if they help with tabular data. The field optimized the house but never tested different engines.

Problem: AdamW is default optimizer for tabular DL
   |
   v
Assumption: New optimizers (Muon, Shampoo, etc.) might work better
   |
   v
Method: Benchmark N optimizers on M tabular datasets with shared protocol
   |
   v
Evidence: Muon consistently outperforms AdamW across datasets
   |
   v
Conclusion: Practitioners should consider Muon as new default

The Increment

One sentence: Before this paper, tabular DL practitioners used AdamW by default without evidence; after, they have systematic benchmarks showing Muon as a stronger choice.

Core Mechanism

The paper doesn’t propose a new optimizer—it’s a benchmarking study. The mechanism is the experimental protocol itself. They take multiple optimizers (AdamW, Muon, Shampoo, Lion, AdamW with EMA, etc.) and run them through identical training pipelines on diverse tabular datasets. Each optimizer gets the same MLP architecture, same hyperparameter search budget, same evaluation metrics. The key is controlling everything except the optimizer, so differences in final performance isolate the optimizer’s contribution.

The protocol has three layers: dataset selection (regression and classification tasks across different domains), model architecture (vanilla MLPs plus variants with different normalization schemes), and hyperparameter tuning (systematic search over learning rates, weight decay, batch sizes). For each optimizer-dataset-architecture combination, they run multiple seeds and report aggregated metrics.

Input: Tabular dataset
   |
   v
[MLP Architecture] <--- Fixed across all experiments
   |
   v
[Optimizer X] <--- Variable being tested
   |    |
   |    +---> Learning rate search
   |    +---> Weight decay search  
   |    +---> Batch size search
   |
   v
[Training Loop] ---> Loss curves, validation metrics
   |
   v
[Aggregation] ---> Mean rank across datasets
   |
   v
Output: Optimizer performance ranking

Think of this like a cooking competition where everyone gets identical ingredients and kitchen equipment, but each contestant uses a different technique to control the heat. You’re not inventing a new stove—you’re systematically testing whether gas, electric, or induction produces better results when everything else (recipe, cookware, timing) is held constant. The “heat control technique” is the optimizer, and the “quality of the dish” is model performance. By running the same recipe (MLP training) with different heat sources (optimizers) across multiple dishes (datasets), you can finally say which technique actually works best, rather than just using whatever your culinary school taught you.

Key Concepts

• Optimizer in deep learning: At its core, an optimizer decides how to update model weights after seeing training data. Imagine you’re trying to find the lowest point in a foggy valley while blindfolded. Each step, you feel the ground’s slope (gradient) and decide which direction to move and how far. AdamW is like having a smart walking stick that remembers recent slopes and adjusts your step size accordingly—if you’ve been going downhill consistently, it takes bigger steps; if the terrain is rough, it takes smaller, cautious steps. Different optimizers are different walking strategies. Muon, for instance, uses momentum in a specific way that helps it navigate the “valleys” of tabular data more efficiently than AdamW’s strategy.

• Exponential Moving Average (EMA) of weights: During training, your model weights jump around as they optimize. EMA is like taking a time-lapse photo with long exposure—instead of using the final frame’s exact position, you blend recent positions together, smoothing out the jitter. Concretely, if your model weight is 0.5 at step 100 and 0.6 at step 101, EMA might give you 0.52 (weighted average favoring recent values). This often produces a more stable, better-performing model than the raw final weights. The paper finds EMA helps AdamW on vanilla MLPs but not consistently across all architectures.

Framework Shift

Before (mainstream approach):        After (this paper):

[Tabular Data]                       [Tabular Data]
      |                                     |
      v                                     v
[MLP Architecture]                   [MLP Architecture]
      |                                     |
      v                                     v
[AdamW] <--- inherited default       [Optimizer Choice] <--- evidence-based
      |                                /    |    \
      v                              /      |      \
[Training]                      [AdamW] [Muon] [Shampoo]...
      |                                     |
      v                                     v
[Model]                              [Benchmark Results]
                                            |
                                            v
                                      [Informed Choice]

From blind inheritance to empirical selection, the core shift is treating optimizer choice as a first-class design decision backed by systematic evidence.

Expert Assessment

Problem choice: This is a real gap, not manufactured. Tabular DL has matured enough that optimizer choice matters, yet the community was running on autopilot. The timing is right—new optimizers exist, tabular benchmarks are standardized, and practitioners need guidance. It’s not glamorous research, but it’s the kind of service work that moves a field forward.

Method maturity: This is solid benchmarking craft. The shared protocol is well-designed: they control for hyperparameter search budget (crucial—some optimizers are just more sensitive to learning rate), use multiple seeds, and test across diverse datasets. The choice to include MLP variants (different normalizations) is smart—it checks whether findings generalize beyond vanilla architectures. No methodological red flags. The simplicity is a feature, not a bug.

Experimental integrity: Baselines are fair. They give each optimizer a proper hyperparameter search, not just default settings. The dataset selection spans regression and classification, different sizes, different domains—good coverage. One minor concern: they don’t deeply investigate *why Muon wins, just that it does. For a benchmarking paper, that’s acceptable, but it leaves mechanistic questions open. The numbers appear solid; they report aggregated ranks rather than cherry-picking individual wins.

Writing quality: The paper is clear and well-organized. The main weakness is in the discussion section—they could dig deeper into when Muon’s advantage is largest (dataset characteristics? architecture details?) and provide more actionable guidance. The EMA findings feel somewhat disconnected from the main Muon narrative; integrating these better would strengthen the story. The related work section is adequate but could better position this against prior optimizer comparisons in other domains.

Verdict: weak accept — Solid empirical work that provides actionable guidance to practitioners, though it doesn’t advance fundamental understanding of optimization.

Takeaways

If you’re training MLPs on tabular data and can afford slightly longer training times, switch from AdamW to Muon—the evidence is clear. More broadly, this paper demonstrates the value of systematic benchmarking in subdomains. The “inherited default” phenomenon happens everywhere: people use whatever worked in the last domain they came from. Before assuming your current optimizer is optimal, check if anyone’s actually tested alternatives in your specific setting. The EMA finding is also portable: if you’re using AdamW on any architecture, try maintaining an EMA of weights during training and using that for inference—it’s a nearly free performance boost on some architectures. Finally, the experimental protocol itself is reusable: if you’re comparing methods, control hyperparameter search budget explicitly, or you’re just measuring sensitivity to tuning rather than true performance.