HyperP: Transferable Hyperparameter Scaling for Stable Language Model Training

Paper: 2603.28743 Authors: Liliang Ren, Yang Liu, Yelong Shen, Weizhu Chen Categories: cs.LG

The Gap

Scaling laws tell us how to grow models efficiently, but they break down when hyperparameters don’t transfer. Existing transfer laws (like μP) work for AdamW but assume first-order optimization. When you switch to newer optimizers like Muon that constrain weights to hyperspheres, the old rules fail. Nobody knew how to transfer learning rates across width, depth, and data scale under hypersphere constraints. Worse, large-scale training still hits instability walls—loss spikes, activation outliers, training collapse—even with careful tuning.

The logical path: If hypersphere optimization structurally prevents certain instabilities, and if we can derive transfer laws for it, then we get both efficiency (tune once, scale everywhere) and stability (no more loss spikes) simultaneously.

Problem: Hyperparameter transfer + stability at scale
    |
    v
Observation: Hypersphere constraints change optimizer dynamics
    |
    v
Method: Derive transfer laws for Frobenius-sphere + Muon
    |
    v
Evidence: Single LR transfers across 6 orders of magnitude
    |
    v
Conclusion: 1.58x efficiency + bounded instability indicators

The Increment

One sentence: Before, you tuned hyperparameters separately at each scale and hoped training stayed stable; after, you tune once at small scale and get guaranteed stability everywhere.

Core Mechanism

HyperP has three components working together. First, it constrains all weight matrices to lie on a Frobenius-norm hypersphere—imagine every weight matrix forced to have exactly the same total magnitude. Second, it uses the Muon optimizer, which updates weights along the sphere’s surface rather than in free space. Third, it derives scaling rules: when you double model width, multiply learning rate by √2; when you add depth, keep per-layer learning rate constant but scale the base rate.

Data flows like this: gradients arrive → Muon projects them onto the tangent space of the hypersphere → updates move weights along the sphere surface → the Frobenius norm stays fixed. For MoE models, they add SqrtGate, which normalizes expert outputs by √(number of experts) to keep signal magnitude constant as you change granularity.

Think of it like training on the surface of a planet. In normal optimization, you can fly anywhere in 3D space—sometimes you drift too high (exploding gradients) or crash into the ground (vanishing gradients). HyperP locks you to the planet’s surface. The Frobenius constraint is the planet’s radius—fixed. Muon is your movement rule: you can only walk along the surface, never jump off. The scaling laws tell you how fast to walk based on the terrain (model architecture). When you encounter a mountain range (deeper model), you don’t speed up—you keep the same walking pace but adjust your base speed. When the planet gets bigger (wider model), you walk proportionally faster because distances scale up. SqrtGate is like adjusting your stride length when the ground splits into multiple paths (MoE experts)—you compensate so your overall speed stays constant.

Key Concepts

• Frobenius-sphere constraint: Imagine you have a matrix of weights—say, a 1000×1000 grid of numbers. The Frobenius norm is just the square root of the sum of all squared entries, like measuring the “total energy” in the matrix. Constraining to a Frobenius sphere means this total energy is fixed—if one weight grows, others must shrink to compensate. Concretely, if your matrix has Frobenius norm 10, and you increase one entry from 0.5 to 0.6, you must decrease other entries slightly so the total norm stays 10. This prevents runaway growth in any direction.

• Hypersphere optimization: Regular gradient descent moves weights in the direction that reduces loss, no restrictions. Hypersphere optimization projects that movement onto the surface of a sphere. Picture a ball: you want to move “downhill” on the loss landscape, but you’re glued to the ball’s surface. So you take the gradient, remove the component pointing radially (toward/away from center), and move along what’s left—the tangent direction. Muon does this projection automatically. The benefit: weight magnitudes can’t explode because they’re locked to the sphere radius.

• μP (maximal update parameterization): This is the insight that learning rates should scale with model width to keep training dynamics similar across scales. Without μP, doubling width means each neuron gets twice as many inputs, so gradients grow by 2×, and you need to shrink learning rate by 2× to compensate. μP bakes this compensation into the parameterization: it scales weight initialization and learning rates together so the “effective” update size stays constant. HyperP extends this to hypersphere constraints, finding that width scaling needs √width factor instead of 1/width because the sphere geometry changes the math.

Framework Shift

Before (AdamW + μP):                After (HyperP + Muon):

Weights in free space               Weights on hypersphere
    |                                   |
    v                                   v
Adam updates                        Muon projects to tangent
    |                                   |
    v                                   v
Scale LR by 1/width                 Scale LR by sqrt(width)
    |                                   |
    v                                   v
Hope for stability                  Guaranteed bounded norms
    |                                   |
    v                                   v
Retune at each scale                Tune once, transfer always

From unconstrained optimization with empirical stability to constrained optimization with provable stability, the core shift is geometry: moving from Euclidean space to a Riemannian manifold changes how scaling laws work.

Expert Assessment

Problem choice: Real gap. Hypersphere optimization is gaining traction (Muon, Shampoo variants), but nobody had transfer laws for it. The instability problem is also genuine—every lab scaling past 10B parameters has war stories about loss spikes. This sits at the intersection of optimization theory and practical engineering, which is where impactful work lives.

Method maturity: Mostly principled. The Frobenius-sphere constraint is clean, and the derivation of scaling laws follows logically from the geometry. The SqrtGate mechanism for MoE feels slightly ad-hoc—it works empirically but lacks the theoretical tightness of the main framework. The “magic exponent” 0.32 for data scaling is borrowed from prior work, not derived here, which is fine but worth noting.

Experimental integrity: Strong baselines. They compare against well-tuned Muon without HyperP, not a strawman. The 1.58× efficiency claim is at 6×10²¹ FLOPs, which is serious compute. Instability metrics (Z-values, activation outliers) are tracked across scales, not cherry-picked. One concern: experiments stop at ~10B parameters; would love to see 100B+ to really stress-test stability claims. The MoE experiments are smaller scale, which is understandable but limits confidence.

Writing quality: The paper front-loads theory and buries practical details. Section 3 (scaling laws) is dense—would benefit from a worked example with actual numbers. The related work section undersells how different this is from μP. Figure 2 (instability metrics) is the paper’s strongest asset but appears late. If I were rewriting, I’d lead with the stability story (show the problem viscerally) then introduce HyperP as the solution.

Verdict: strong accept — Solves a real problem with principled methods, delivers both efficiency and stability, and opens a new research direction for hypersphere optimization at scale.

Takeaways

Steal the hypersphere constraint idea even if you’re not scaling LLMs. Any time you’re training something with matrix weights and hitting instability, try constraining Frobenius norms—it’s a simple projection operation. The SqrtGate normalization (divide by √N when splitting into N branches) is a general trick for MoE or ensemble methods to keep signal magnitude constant. The broader lesson: when switching optimizers, don’t assume old scaling laws transfer—rederive them from the optimizer’s geometry. Finally, the experimental protocol of tracking multiple instability indicators (not just loss) across scales is worth copying for any large-scale training work.