Quantization Damage Is Diffuse, So Spend the Next Bit Globally

Paper: 2609.01587 Authors: Jundong Hu, Shekar Ramachandran Categories: cs.CL, cs.LG

The Gap

Post-training quantization is standard practice for serving large models cheaply, and its accuracy cost is uneven — some layers tolerate low precision, others do not. The standard response is to tune per model: find the sensitive layers, protect them, quantize the rest harder. That practice requires knowing where the damage is.

The paper shows that the natural ways of answering “where” do not work. Three intuitive hypotheses are tested — that damage lives in task circuits (where the model computes), that it lives in weight statistics, and a third of the same family — and none of them predicts which layers benefit from restored precision. The signals correlate with damage, which is presumably why the hypotheses seemed reasonable, but correlation is not the same as identifying where spending precision actually helps.

That distinction is the paper’s methodological core, and it is a specific one: a cheap signal that tells you a layer is damaged does not tell you that repairing it recovers accuracy. Only intervention answers that.

   THE STANDARD PRACTICE, AND THE ASSUMPTION UNDER IT

   PTQ is standard; accuracy cost is UNEVEN
     some layers tolerate low precision, others do not
        |
        v
   response: TUNE PER MODEL
     find sensitive layers -> protect them -> quantize the rest harder
        |
        v
   which REQUIRES knowing WHERE the damage is

   THREE INTUITIVE HYPOTHESES, ALL TESTED
     damage lives in TASK CIRCUITS (where the model computes)?
     damage lives in WEIGHT STATISTICS?
     (and a third of the same family)
        |
        v
   [GAP] NONE predicts which layers benefit from restored precision
     the signals DO correlate with damage -- which is why the
     hypotheses seemed reasonable -- but correlation is not
     identification of where precision SPENDING helps

The Increment

One sentence: Before this paper, layer sensitivity was inferred from cheap signals; after it, causal intervention shows recovery is diffuse, and at a matched budget global granularity beats selectively repairing the most recoverable layers by 21-52 points.

Core Mechanism

The ground truth is established by causal mixed-precision intervention: raise each layer to 8-bit in turn and measure the accuracy it recovers. That is the right instrument, and it is expensive — which is precisely why practitioners reach for cheap proxies. The measurement spans 9 open-weight models in 4 architecture families, enough for the conclusion to be about quantized models rather than about one.

Recovery is diffuse. For 8 of 9 models, recovering 75% of the gap takes roughly half the layers. So the damage is not concentrated in a few culprits you could protect; it is spread across the model, and any per-layer protection schedule is working against the grain of the actual distribution. The lone exception is Qwen3-8B, which is sharply concentrated — a useful reminder that the phenomenon is not universal, and a case the paper keeps visible rather than averaging away.

Then the result that turns the diagnosis into a prescription. At a matched precision budget — the comparison that matters, since precision is the resource — spending it globally on finer quantization granularity beats locally repairing the most recoverable layers, for all 8 group-128-compatible models, by 21-52 points, including the concentrated Qwen3-8B. Two things about that last clause are notable. The margin is large. And the globally-spent budget wins even on the model where damage is concentrated, which is the case where the local-repair argument ought to be strongest. The one excluded model, OpenLLaMA, is excluded for a mechanical reason — its width rules out group-128 — not because it disagreed.

Two secondary findings sharpen the picture. The residual is budget-limited: 8-bit is near-lossless across RTN, GPTQ and AWQ, so there is a ceiling on what more precision can buy. And the location of peak recovery correlates with architecture within a family, though not across families — a hint of structure that stops short of a predictive rule, which is consistent with the headline negative result.

   GROUND TRUTH: CAUSAL MIXED-PRECISION INTERVENTION
     raise each layer to 8-bit in turn,
     measure the accuracy it RECOVERS
     across 9 open-weight models / 4 architecture families
        |
        v
   FINDING 1: RECOVERY IS DIFFUSE
     8 of 9 models: recovering 75% of the gap takes ~HALF the layers
     exception: Qwen3-8B is SHARPLY CONCENTRATED
     -> per-layer protection works against the actual distribution
        |
        v
   FINDING 2 (MATCHED PRECISION BUDGET)
     spend GLOBALLY on finer quantization GRANULARITY
       vs
     locally repair the MOST RECOVERABLE layers
        -> global wins by 21-52 POINTS, for all 8
           group-128-compatible models
        -> INCLUDING the concentrated Qwen3-8B
           (where local repair should have looked best)
     (OpenLLaMA excluded: its width rules out group-128)

   SECONDARY
     residual is BUDGET-LIMITED: 8-bit is near-lossless
       across RTN, GPTQ, AWQ
     peak-recovery location correlates with architecture
       WITHIN a family, but NOT across families
        -> a hint of structure, not a predictive rule

Think of it as restoring an old photograph by re-scanning it at higher resolution rather than hand-repairing the worst scratches. The scratch-repair instinct says: find the ugliest blemishes and fix those first. But if the actual loss is fine grain — a slight softening spread evenly across the whole image — then repairing fifty scratches leaves the softness untouched, while a higher-resolution scan sharpens everything at once. The paper’s measurement is the equivalent of first establishing, empirically, which regions would improve if re-scanned — and finding that for eight of nine photographs the improvement is spread across roughly half the frame, and that the even re-scan beats targeted repair even on the one photograph that was genuinely concentrated.

Key Concepts

• Causal intervention as ground truth: raising each layer to 8-bit and measuring recovered accuracy, rather than inferring sensitivity from cheap statistics. It is expensive by design, and the paper’s point is that the cheap alternatives do not identify where spending precision helps.
• Diffuse recovery: the finding that ~75% of the recoverable gap takes about half the layers, for 8 of 9 models. It explains why per-layer protection schedules underperform — they assume a concentration the distribution does not have.
• Matched-budget comparison: comparing global granularity against local repair at equal precision. Precision is the resource being allocated, so a comparison at unequal budgets would not answer the allocation question.
• Budget-limited residual: 8-bit being near-lossless across three quantization methods. It bounds the value of additional precision and frames the whole result as an allocation problem within a ceiling.
• Correlation is not identification: cheap signals correlate with quantization damage but do not predict where restoration helps. It is the paper’s generalisable methodological warning.

Framework Shift

Before (infer sensitivity, protect the worst layers):
  cheap signals correlate with damage
  -> pick sensitive layers, protect them, compress the rest
  -> implicitly assumes damage is CONCENTRATED
  -> per-model tuning is the norm

After (intervene, then allocate globally):
  causal intervention as ground truth
  -> recovery is DIFFUSE in 8 of 9 models
  -> at a matched budget, GLOBAL granularity beats local repair
     by 21-52 points, even on the concentrated model
  -> 8-bit residual is near-lossless: the budget is capped

From protecting the layers that look damaged to allocating precision where it demonstrably recovers accuracy, the core shift is that identifying damage and identifying where repair pays are different questions, and only intervention answers the second.

Expert Assessment

Problem choice: Excellent, and it attacks a practice rather than a technique. Per-model quantization tuning is widespread and rests on an assumption about where damage lives; testing that assumption with the right instrument, and finding it false, is more valuable than another quantization method.

Method maturity: The design is exactly right for the claim: causal intervention as ground truth, nine models across four families, and a matched-budget comparison, since precision is the resource. Reporting the lone concentrated exception and keeping it in the headline result — the global approach winning even there — is what makes the conclusion robust rather than an average that hides a counterexample. The two secondary findings are genuinely secondary and are treated that way.

Experimental integrity: The comparisons are well specified: group-128 compatibility is stated as the reason for excluding OpenLLaMA, and the margin is reported as a range across models rather than as a single flattering number. The result that cheap signals mislead is presented as a methodological finding with wider scope than quantization, and the paper is careful to say the signals correlate — it does not claim they are meaningless. The scope limit is that this is the PTQ setting with a small additional budget, and the authors say so; the conclusion is about which default wins within a budget, not about quantization in general.

Writing quality: The title states the prescription and the abstract supplies the evidence, which is a good pairing for a paper whose contribution is an allocation rule. One addition would make it more usable: a concrete comparison of what “global granularity” means in practice for the models tested, since the reader’s actionable question is which knob to turn rather than the direction to turn it.

Verdict: strong accept — it tests a widely assumed premise with the instrument that can settle it, finds the premise false, and converts the diagnosis into a budget allocation rule with a substantial margin.

Takeaways

• Distinguish diagnosing damage from locating where repair pays. Cheap signals can be correlated with a problem without telling you where spending a fixed budget helps.
• Check whether damage is actually concentrated before building a protection schedule. Here it was diffuse in 8 of 9 models, so per-layer protection was working against the distribution.
• Compare allocation strategies at a matched budget. If precision is the resource, a comparison at unequal budgets answers a different question.
• Watch the exceptions rather than averaging them out. The global approach winning even on the one concentrated model is what makes the rule safe to apply by default.