Quantized LLMs Aren't What They Seem: Why Accuracy Fails and What to Measure Instead

Paper: 2607.08734 Authors: Baha Rababah, Cuneyt Gurcan Akcora, Carson K. Leung Categories: cs.AI

The Gap

Existing research on post-training quantization (PTQ) has reached a clear boundary: we know it makes models smaller and faster, and we evaluate its success almost entirely by tracking accuracy (like MMLU scores) and perplexity. The specific limitation this paper addresses is that these metrics are a dangerous illusion. They can show a quantized model performing “just as well” as its full-precision counterpart, while its internal decision-making process has fundamentally and unpredictably changed. The gap is the lack of a behavioral evaluation framework for quantization.

Problem: PTQ enables deployment in resource-constrained environments.
   |
Assumption: Accuracy and perplexity are sufficient metrics to evaluate quantization quality.
   |
Logical Path:
   |
   +-> If accuracy is preserved, model behavior is preserved. (Old belief)
   |
   +-> But: A model can get the same final score by answering a *different* set of questions correctly.
   |
   v
New Method: Introduce "Correctness Agreement" (CA) metric + Statistical analysis of weight distortions.
   |
   v
Evidence: Experiments on multiple models (LLaMA, Mistral, etc.) with GPTQ/AWQ from 8-bit to 2-bit.
   |
   v
Conclusion: Behavioral divergence emerges *before* accuracy drops. Query/Key projections are most sensitive. Evaluation needs a paradigm shift.

The Increment

One sentence: Before this paper, we judged quantized models by *how much they knew; after this paper, we must also judge them by how they think, revealing that ostensibly equivalent models can be fundamentally different decision-makers.

Core Mechanism

The paper’s method works in two coupled layers: a new evaluation metric and a diagnostic toolkit.

First, for any set of questions, you run the base model (e.g., LLaMA-13B FP16) and a quantized version (e.g., LLaMA-13B GPTQ-4bit). You record which questions each gets right. The Correctness Agreement (CA) is simply the percentage of questions both models answer correctly, divided by the total number of questions at least one model answers correctly. This isolates behavioral overlap from absolute performance.

Second, to explain why CA drops, the authors treat quantization not just as a computational shortcut, but as a structural operator applied to the model’s attention weights (W_Q, W_K, W_V, W_O). They measure the distortion (using metrics like Wasserstein distance, cosine similarity, Frobenius norm) on each weight matrix before and after quantization. By correlating these distortions with CA, they identify which components (like Query and Key projections) are most fragile.

Inputs:
  +--> Base Model (M_fp) --+--> Predictions_fp --> Correctness_fp
  |                        |
  +--> Quantized Model (M_q) -> Predictions_q --> Correctness_q
                                                     |
   [Correctness Agreement = |Correct_fp & Correct_q| / |Correct_fp U Correct_q|]
                                                     |
   +--> Statistical Analysis Layer
   |    - Compute W_fp and W_q for each layer/projection.
   |    - Measure distortion: Cosine Sim, Wasserstein, L2 Norm.
   |
   v
   [Output: Correlation maps: Layer-wise distortion vs. CA drop]
   [Insight: Q/K projections are most sensitive; distortion is non-linear at low bits.]

Let’s use a medical check-up metaphor. Imagine the base model is a healthy person (Patient A) and the quantized model is their “optimized” twin (Patient B) who’s been put on a strict, simplified diet.

• Old evaluation (Accuracy/Perplexity) is like checking only height and weight. Patient B might have the exact same numbers as Patient A. “They’re equivalent!” you’d say.
• New evaluation (Correctness Agreement) is like a full neurological and cognitive test. You give them both the same complex puzzles. You find that while their total score is identical, Patient B solves the puzzles using completely different strategies, and fails on a subset Patient A finds easy. Their *process has diverged.
• The diagnostic toolkit (Weight Analysis) is the blood work and MRI. You see that Patient B’s blood sugar and cholesterol (the quantized Q/K weights) are dramatically altered, explaining the strange cognitive test results. The V/O weights are more stable. The diet (quantization) didn’t just make them lighter; it changed their internal biochemistry in a non-obvious way.

This metaphor is load-bearing because it captures the core argument: a superficial equivalence (same weight/height) masks a profound operational divergence (different cognitive strategy and blood chemistry), and you need the right tests to see it.

Key Concepts

• Correctness Agreement (CA): Forget technical definitions. Imagine two students take a 100-question test. Student A scores 85, Student B scores 85. They’re “equal.” But Student A got questions 1-85 right. Student B got questions 16-100 right. Their correctness agreement is only 70% (the overlap of 16-85). The 15 points they each lost were on completely different questions. CA measures this overlap in *success, exposing that their knowledge structures, while maybe equally deep, are built on different foundations. This is the key insight.
• Quantization as a Structural Operator: Quantization isn’t just “rounding numbers to save space.” It’s a drastic transformation of the model’s internal geometry. Think of the attention weights (the model’s “thinking pathways”) as a intricate map of roads. Full precision is a detailed map with every alleyway. Quantization (to 4-bit, 2-bit) is like replacing that map with a simplified cartoon version that only shows highways. You might still get from City A to City B (correct answer), but the route is completely different, and many small towns (subtle capabilities) become inaccessible or are routed through completely wrong areas. The paper measures how much this “cartoonification” distorts the original map’s structure.

Framework Shift

Before (mainstream approach):          After (this paper):
+-----------------------+             +-----------------------+
| Model                 |             | Model                 |
|   |                   |             |   |                   |
|   v                   |             |   v                   |
| Run on Benchmark      |             | Run on Benchmark      |
|   |                   |             |   |                   |
|   v                   |             |   v                   |
| Get Score (Accuracy)  |             | Get Scores (Accuracy) |
|   |                   |             |   +---------+         |
|   v                   |             |             |         |
| DONE: "Model is OK"   |             |             v         |
+-----------------------+             | Compare Predictions   |
                                      |   |                   |
                                      |   v                   |
                                      | Compute CA &          |
                                      | Weight Distortion     |
                                      |   |                   |
                                      |   v                   |
                                      | DONE: "Here's *how*   |
                                      | it differs"           |
                                      +-----------------------+

From score-centric equivalence to process-centric divergence, the core shift is recognizing that identical performance metrics can stem from fundamentally different internal mechanisms.

Expert Assessment

Problem choice: This is a real and important gap. The field has been almost complacent with quantization evaluation, accepting perplexity as gospel. This paper correctly identifies that deploying a model isn’t just about its peak performance, but about its reliability and predictability. It’s a timely critique as quantization moves from research to massive production.

Method maturity: It’s a clever and insightful approach. The CA metric is simple, interpretable, and directly addresses the problem. Using established statistical distances to measure weight distortion is methodologically sound. It’s not brute force; it’s a precise diagnostic tool applied to a under-studied phenomenon. A simpler approach might be to just look at per-question accuracy flip rates, but CA and the weight analysis provide a much richer narrative.

Experimental integrity: The baselines (FP16 base models) are fair. Testing across multiple model families (LLaMA, Mistral, Phi) and quantization methods (GPTQ, AWQ) strengthens the claim. The numbers—showing CA dropping significantly even with high accuracy—are convincing. A potential red flag: the analysis is primarily on English, comprehension-heavy tasks. Does this divergence manifest differently in generative, multilingual, or code tasks? The paper could have been stronger by exploring this.

Writing quality: The paper is clear and well-structured. The biggest corner cut is in the related work section, which could better position this work within the broader literature on model robustness and fairness, where similar “performance illusion” themes appear. Rewriting the introduction to more viscerally illustrate a real-world failure case of this illusion (e.g., a quantized medical model agreeing on 95% of diagnoses but diverging critically on 5%) would elevate the impact.

Verdict: weak accept — It introduces a simple, powerful lens (CA) for a critical real-world problem, backed by solid experiments. It’s not the final word, but it successfully opens a new and necessary line of inquiry.

Takeaways

1. Steal the “Correctness Agreement” mindset. Whenever you evaluate a model modification (quantization, distillation, pruning, fine-tuning), don’t just track overall accuracy. Always measure the *agreement in correct predictions between the original and modified model. This is a cheap, powerful sanity check for behavioral drift.
2. Quantize with a layer-sensitive strategy. The finding that Q/K projections are more sensitive than V/O suggests you could apply mixed-precision quantization—spending more bits on Q/K and fewer on V/O—for a better efficiency/faithfulness trade-off. Don’t treat all weights equally.
3. Beware the “accuracy plateau.” When optimizing a model, seeing accuracy plateau and stabilize can be misleading. This paper shows that the model’s decision boundaries can still be shifting wildly beneath that plateau. Use behavioral metrics to probe stability.