Learning Between the Peaks: How Anisotropy Reshapes Kernel Ridge Regression

Paper: 2608.28564 Authors: Lorenzo Rizzi, Arie Wortsman Zurich, Bruno Loureiro Categories: stat.ML, cs.LG

The Gap

The theory of kernel ridge regression has a canonical picture of how learning proceeds with sample size, and its most striking feature is the peaks: the variance term has local maxima at integer sample complexities. These are not artefacts — they are the multiple-descent phenomenon, where more data momentarily hurts, and they are one of the more counterintuitive predictions in the theory.

That picture assumes isotropy — data whose covariance is flat across directions. Real data is rarely isotropic. The natural generalisation is a power-law anisotropic covariance, decaying with an exponent, and the question is what happens to the canonical picture when you stop assuming flatness.

This is not a perturbation question with a small answer. Anisotropy changes the effective dimensionality of the problem, and effective dimension is what governs how much a fixed number of samples can teach you. So the learning curve should be reshaped rather than merely shifted, and the reshaping is what the paper derives.

   THE CANONICAL PICTURE, AND THE ASSUMPTION UNDER IT

   kernel ridge regression, isotropic data:
     variance has LOCAL MAXIMA at INTEGER sample complexities
       -> the "peaks": more data momentarily HURTS
       -> multiple descent, and a genuinely counterintuitive
          prediction of the theory
        |
        v
   ASSUMPTION: ISOTROPY -- covariance flat across directions
        |
        v
   real data: rarely isotropic
     natural generalisation: POWER-LAW ANISOTROPIC covariance
                             decaying with an exponent
        |
        v
   [GAP] what happens to the canonical picture when
         flatness is dropped?
        |
        v
   anisotropy changes the EFFECTIVE DIMENSIONALITY,
   which is what governs how much a given number of
   samples can teach
        -> expect RESHAPING, not a shift

The Increment

One sentence: Before this paper, the kernel ridge regression learning curve was characterised for isotropic data; after it, sharp asymptotics under power-law anisotropy show the peaks damping with weak anisotropy, bias transitioning at fractional complexities, and variance becoming sample-size-independent under strong anisotropy.

Core Mechanism

The setup pins down the regime precisely: anisotropic Gaussian data whose input covariance decays as a power law with exponent α at least zero, with polynomial inner-product kernels, in the polynomial high-dimensional regime where the sample size scales as a power of the dimension. Sharp expressions follow for the kernel spectrum and the generalisation error, and the results split cleanly by the strength of anisotropy.

Weak anisotropy, the exponent between zero and one. The problem remains effectively high-dimensional and retains some features of the isotropic case while departing in others. Two departures matter most, and they pull in opposite directions:

• The variance still peaks at integer sample complexities, but the peaks are progressively damped as the exponent grows. So multiple descent survives; it just weakens. The counterintuitive phenomenon is not an artefact of isotropy, and it fades smoothly rather than disappearing.
• For targets strongly aligned with the data’s principal directions, the bias drops at fractional sample complexities. This is the more surprising result, because it decouples the bias transitions from the interpolation peaks. In the isotropic picture the notable events happen at integers; here the bias makes its moves at non-integers, so the two curves’ features are no longer in register. Anyone reading a learning curve for integer-valued events would miss the bias transitions entirely.

Strong anisotropy, the exponent above one. This is a qualitative change rather than more of the same: the effective dimension of the problem is constant, and consequently the variance stops depending on sample size altogether. It either plateaus under ridgeless interpolation or vanishes at an explicit rate under a fixed ridge penalty. A learning curve whose variance is flat in sample size is not a shape the isotropic theory produces at all.

The bias transition becomes sharp. Under strong anisotropy, the bias transition is governed by the target’s decay rate: below a threshold, learning is abrupt rather than gradual; above it, the bias decays as a power law that recovers the classical source and capacity rates. So strong anisotropy separates two regimes, one of which looks like the familiar theory and one of which does not.

Finally, the results are specialised to single-index targets, where the alignment of the index with the data’s principal directions determines how much anisotropy matters. That specialisation is the practically useful part: it names a measurable property of the target that tells you which regime you are in.

   THE RESULTS, SPLIT BY ANISOTROPY STRENGTH

   SETUP: anisotropic Gaussian data, covariance ~ power law (exponent a >= 0)
          polynomial inner-product kernels
          polynomial high-dimensional regime: n scales as a power of d
          -> sharp expressions for the kernel SPECTRUM and the ERROR

   WEAK ANISOTROPY (0 < a < 1)
     still effectively high-dimensional
     variance STILL PEAKS at integer complexities,
       but peaks are PROGRESSIVELY DAMPED as a grows
       -> multiple descent survives, weaker
     for targets ALIGNED with principal directions:
       bias drops at FRACTIONAL sample complexities
       -> decouples bias transitions from the peaks
       -> integer-indexed reading of the curve misses them

   STRONG ANISOTROPY (a > 1)
     effective dimension is CONSTANT
     -> VARIANCE STOPS DEPENDING ON SAMPLE SIZE
          plateaus under ridgeless interpolation
          vanishes at an explicit rate under fixed ridge
     BIAS transition is SHARP, governed by the target's decay:
       below a threshold: learning is ABRUPT, not gradual
       above it: power-law decay, recovering the classical
                 source and capacity rates

   SPECIALISATION: single-index targets
     the ALIGNMENT of the index with the data's principal
     directions determines how much anisotropy matters

Think of it as sampling a landscape that is a long ridge rather than a field of hills. If the terrain is isotropic — hills everywhere — then adding samples fills in the map uniformly, and the theory’s peculiar prediction is that at certain sample counts your estimate temporarily gets worse before improving. Now make the terrain a long, narrow ridge: the interesting variation runs along one direction and almost nothing varies across it. Adding samples mostly tells you more about the direction you already understood, and once the cross-direction is pinned down, more samples add little — which is the constant effective dimension and the sample-independent variance. The fractional-complexity bias result is the equivalent of discovering that the ridge’s features only become resolvable at sample counts that are not round numbers, so someone sampling at the conventional checkpoints would simply never see them.

Key Concepts

• Power-law anisotropy as the generalisation that matters: covariance decaying with an exponent rather than being flat. It is the minimal assumption that captures real data’s geometry, and the paper’s results parameterise the whole family by one number.
• Peaks damped, not removed: multiple descent survives weak anisotropy but weakens progressively. It means the phenomenon is a property of learning under this kernel family rather than an artefact of the isotropic special case.
• Decoupled transition locations: bias moves at fractional complexities while variance peaks at integer ones. The practical consequence is significant — a learning curve read at integer checkpoints will systematically miss one of the two transitions.
• Constant effective dimension under strong anisotropy: the reason variance becomes independent of sample size. It is a qualitative regime change, and it identifies the condition under which more data stops helping the variance term at all.
• Alignment as the controlling measurable: the alignment of a single-index target with the data’s principal directions determines the effect. It is the quantity to estimate if you want to know which regime your problem is in.

Framework Shift

Before (isotropic theory):
  variance peaks at integer sample complexities
  bias transitions read off the same integer scale
  -> one canonical picture, and one regime of behaviour

After (power-law anisotropy, parameterised by one exponent):
  WEAK (0 < a < 1):
    peaks survive but DAMP; bias moves at FRACTIONAL points
  STRONG (a > 1):
    effective dimension CONSTANT
    variance INDEPENDENT of sample size
    bias transition SHARP, governed by the target's decay rate
  -> a family of regimes, selected by the data's geometry
     and the target's alignment

From one canonical learning curve to a family selected by the data’s geometry and the target’s alignment, the core shift is that where the interesting events sit on the sample-size axis depends on how anisotropic your data is.

Expert Assessment

Problem choice: Good, and it targets the assumption most obviously in need of relaxation. Isotropic theory is clean and its predictions — multiple descent especially — are widely cited, so establishing how they fare under a realistic covariance is a well-motivated question rather than a technical variation.

Method maturity: The results are sharp asymptotics, and the regime is stated precisely — polynomial high-dimensional, sample size scaling as a power of the dimension, polynomial kernels, power-law covariance. That precision is what makes a family of conclusions possible rather than one. The split by anisotropy strength is the organising insight: weak anisotropy reshapes within the existing picture while strong anisotropy changes the picture qualitatively, and identifying which of the two you are in is exactly the practical question.

Experimental integrity: These are theoretical results, so the scrutiny is on assumptions and on whether the claimed sharpness is genuine. The power-law covariance and polynomial kernel are modelling choices that make the analysis tractable, and the honest caveat is that real data rarely has exactly this structure — the value of the results is as a caricature that isolates the mechanism. The single-index specialisation is the part that connects to practice, since alignment is estimable, and it would be worth extending further.

Writing quality: The division into weak and strong anisotropy cases is stated early and organises everything after it, which is the right structure for a result with two regimes. The most valuable addition would be a figure: the learning curve for a few values of the exponent, with the peaks damping and the fractional bias drops appearing, since the whole finding is about the shape of a curve and is far easier to see than to read.

Verdict: strong accept — a sharp characterisation of how a realistic data geometry reshapes the canonical kernel learning curve, including a regime in which more data stops reducing variance entirely.

Takeaways

• Check your data’s anisotropy before trusting learning-curve predictions. The locations of the interesting transitions on the sample axis depend on the covariance’s decay.
• Do not read a learning curve only at integer sample counts. Under weak anisotropy the bias transitions happen at fractional complexities and will be invisible at conventional checkpoints.
• Interpret a flat variance term as a geometry signal, not a plateau. Constant effective dimension makes more data unhelpful for variance, which is a different situation from convergence.
• Estimate target alignment to know which regime you are in. How much anisotropy matters depends on how well the target lines up with the data’s principal directions.