How Much Rank Does LoRA Need? A Task-Dependent Answer

Paper: 2608.26052 Authors: Gerard Conangla Planes Categories: cs.AI, cs.CL, cs.LG

The Gap

LoRA rank is chosen empirically. You pick a value, train, evaluate, and try again — 8, 16, 32, 64 — until the returns flatten. That practice works and says nothing about why a given rank is enough, or whether the answer depends on the task. There is no task-dependent account of the approximation error achievable at each rank.

The distinction matters because “rank” is doing two different jobs that are easy to conflate. A LoRA update has to approximate the attention function — the probability distribution over tokens it ultimately produces — and it also has to approximate the finite attention scores that give rise to it. These are not the same target, and the rank required for one need not equal the rank required for the other. Without a theory separating them, a rank sweep cannot tell you whether you are limited by capacity or by a mismatch between what you are matching and what you care about.

   HOW LORA RANK IS CHOSEN TODAY
   pick r in {8, 16, 32, 64} -> train -> evaluate -> repeat
        |
        v
   conclusion: "the returns flattened at r = X"
        |
        v
   [UNANSWERED]
     - is X task-dependent?
     - what error is achievable at each r?
     - are we limited by capacity, or by matching
       the WRONG TARGET?
        |
        v
   [TWO TARGETS, EASILY CONFLATED]
     attention FUNCTION (the probabilities)   <- what you want
     attention SCORES   (finite logits)       <- what you fit
        |
        v
   [GAP] no task-dependent theory of the error
         achievable at rank r

The Increment

One sentence: Before this paper, LoRA rank was a swept hyperparameter; after it, a task-dependent bound on the expected error at each rank separates the rank needed to match attention probabilities from the rank needed to match attention scores, and shows the first can be strictly smaller.

Core Mechanism

The theory fixes three things and then bounds the error. A pretrained attention head, a target attention function, and a distribution over inputs from the downstream task. With those fixed, the question becomes concrete: what is the smallest expected KL error achievable by a rank-r query LoRA update? Note that the input distribution is part of the setup, which is what makes the result task-dependent rather than architecture-dependent.

The bounds come in a matched pair, and their shape is informative.

When the target attention probabilities are bounded away from zero, there is a lower bound on the error proportional to psi(||d||_2), where d is the difference between candidate and target attention scores and psi(t) = min{t^2, t}. That function is doing real work: it is quadratic for small disagreement and linear for large disagreement. So the error is not simply quadratic in score error — once the scores are far enough apart, the relationship flattens. Matching it is an unconditional upper bound of min{||d||_2^2 / 4, sqrt(2) * ||d||_2}, the same two-regime shape.

Under explicit realizability, geometry and moment conditions, the best rank-r error is then sandwiched between an explicit multiple of psi(sqrt(T_r)) and min{T_r / 4, sqrt(2 T_r)}, where T_r is the downstream-weighted tail energy of the target update. That quantity is the practical heart of the result. T_r is not raw spectral energy — it is weighted by the downstream input distribution, so the rank you need depends on which parts of the spectrum the task actually exercises. A direction the task never probes costs no rank to neglect. This is the formal version of the intuition practitioners have been acting on without a justification.

The paper then builds a case that separates the two targets. It constructs explicit families in which softmax saturation makes the rank required to match the attention function strictly smaller than the rank required to match the finite logits. The reasoning is intuitive once stated: logits can grow without bound while the resulting probabilities saturate, so above a certain scale, differences in logits stop changing the distribution. A rank budget spent matching logits faithfully can be partly wasted on distinctions the softmax erases. This is the concrete payoff of not conflating the two targets.

Two extensions round it out. Target-Fisher bounds apply when candidate scores remain within a fixed range of the target scores — a local rather than global account. And there is an unrestricted lower bound when a subset of tokens carries most of the probability mass, which is the regime attention actually operates in. Finally, the analysis extends to fused multi-head LoRA and joint query/key updates, exposing the effects of rank sharing across heads and of the query/key factorization constraint.

   SETUP: fix three things, then bound the error
     pretrained attention head
     target attention function
     input distribution from the DOWNSTREAM TASK
        |
        v
   ERROR AT RANK r (expected KL)
     lower bound ~ psi(||d||_2),   psi(t) = min{t^2, t}
     upper bound   min{||d||_2^2 / 4, sqrt(2) ||d||_2}
        |
        v
   UNDER REALIZABILITY / GEOMETRY / MOMENT CONDITIONS
     best rank-r error between
        multiple of psi(sqrt(T_r))  and  min{T_r/4, sqrt(2 T_r)}
     T_r = DOWNSTREAM-WEIGHTED TAIL ENERGY of the target update
        -> the task decides which spectral directions cost rank
        |
        v
   SEPARATING THE TWO TARGETS
     match attention FUNCTION  -> rank r1
     match finite LOGITS       -> rank r2
     softmax saturation can give  r1 STRICTLY SMALLER than r2
        -> a faithful logit match can spend rank on
           distinctions the softmax erases

   EXTENSIONS: target-Fisher bounds (local), unrestricted lower
   bound under concentrated probability mass, fused multi-head
   LoRA, joint query/key updates

Think of it as deciding how much detail a map needs by asking where you will actually drive. A cartographer could try to reproduce the terrain exactly — every contour, every ditch — and spend enormous effort on fidelity. But the map only needs the detail that changes your route. A region you will never enter costs nothing to draw coarsely, no matter how complicated it is. T_r is precisely this: the tail energy of the target update weighted by the downstream input distribution, so complicated spectral directions your task never exercises are cheap to neglect. The softmax-saturation result is the second half of the same idea: below a certain point, extra precision in the coordinates changes nothing about which road you take, so paying for it is pure waste.

Key Concepts

• T_r, downstream-weighted tail energy: the quantity that makes the bound task-dependent. It is what remains of the target update’s spectrum after weighting by the input distribution, so it says which spectral directions the task actually uses — and therefore which ones cost rank to approximate.
• The two-regime psi(t) = min{t^2, t} shape: quadratic for small score error, linear for large. The practical reading is that error grows more slowly than a naive quadratic argument would suggest once disagreements become large, and that bounds derived from a purely quadratic assumption will misstate the achievable error.
• Attention function versus finite logits: two different approximation targets, and the softmax can make the first strictly cheaper to match. It means a rank sweep is measuring the cost of one target while a practitioner may care about the other.
• Rank sharing and query/key factorization: the constraints that appear once the analysis moves from a single query update to fused multi-head LoRA and joint updates. These are where a per-head rank choice stops being independently optimizable.

Framework Shift

Before (rank as a swept hyperparameter):
  try r = 8, 16, 32, 64 -> pick where returns flatten
  -> no task dependence in the answer
  -> no separation of "capacity" from "wrong target"

After (rank bounded by a task-dependent quantity):
  error at rank r bounded below by psi(||d||_2)
  best rank-r error governed by T_r, the
    DOWNSTREAM-WEIGHTED tail energy
  -> directions the task does not exercise are cheap
  -> softmax saturation can make the rank needed
     to match the FUNCTION smaller than for the LOGITS

From treating rank as a knob to be swept, to bounding the error it can achieve given a task, the core shift is that the input distribution — not just the architecture — determines how much rank is enough.

Expert Assessment

Problem choice: Good and genuinely useful. LoRA rank is one of the most-typed hyperparameters in applied work and one of the least theorised, and asking for a task-dependent bound rather than a universal rule is the right formulation — a universal answer would likely be useless.

Method maturity: This is a theory paper and it is constructed carefully. Fixing the input distribution as part of the setup is the decision that makes the result task-dependent, and the two-regime psi shape from both sides (lower bound and matching upper bound) gives the result teeth rather than leaving it as an asymptotic remark. The softmax-saturation construction is the most interesting part, because it is a case where the theory says something a sweep cannot: two targets with genuinely different rank requirements, where the one you likely care about is the cheaper one.

Experimental integrity: There are no experiments, and the claims are theorems under stated conditions. The relevant scrutiny is therefore the conditions — realizability, geometry, moment assumptions, and scores-remaining-within-a-range for the Fisher bounds. Each is stated rather than hidden, which is the right practice, but a practitioner needs to judge how far their fine-tuning setup sits from those assumptions before applying the bounds.

Writing quality: The paper’s structure — setup, bounds, T_r, the separation of targets, then extensions — is logical and the abstract states the bound shapes explicitly rather than gesturing at them. What would help a practical reader most is a worked numerical illustration: a plausible attention head, a computed T_r, and the implied rank, so that the theory can be compared against the sweep people run today.

Verdict: accept — a careful task-dependent theory for a hyperparameter that is usually set by habit, with the finiteness of the score approximation properly accounted for rather than assumed away.

Takeaways

• Judge LoRA rank against the task, not against the architecture. The quantity that governs the achievable error is the downstream-weighted tail energy, so ranks that are right for one task can be wasteful or insufficient for another on the same model.
• Do not assume the rank needed to match attention probabilities equals the rank needed to match the logits. Softmax saturation can make the functional target strictly cheaper, so a rank sweep may be measuring the wrong thing.
• Expect sub-quadratic error growth in large-disagreement regimes. Error scales as min{t^2, t} in the score gap, so quadratic intuitions will overstate the cost once the gap is wide.
• Be explicit about which of the two targets you are evaluating. If your evaluation only checks logits, you may be paying rank to match distinctions that never reach the output distribution.