Where Post-Training Quantization Breaks Text Embedders

Paper: 2609.16391 Authors: Hyojung Han Categories: cs.IR, cs.CL

The Gap

Weight-only post-training quantization is the cheapest way to shrink a retrieval embedder. The playbook that people actually use was carried into LLM quantization largely intact: protect the embedding table, allocate bits by module sensitivity, prefer a ranking-aware objective over weight reconstruction.

The paper tests that advice on retrieval embedders directly, not on generative LMs. The gap is not “nobody quantized embedders”; it is that every heuristic fails to transfer as stated. Advice that sounds universal is, under measurement, architecture- and bit-width-specific — and sometimes inverted.

   RECEIVED PTQ ADVICE VS RETRIEVAL REALITY

   advice (from LLM practice)
     1. protect the embedding table
     2. allocate bits by module sensitivity
     3. ranking-aware objective > reconstruction
         |
         v
   test on 5 checkpoints / 4 embedder families
     grid of bit widths and group sizes
     isolate embedding, attention, FFN at each width
         |
         v
   EVERY HEURISTIC FAILS TO TRANSFER AS STATED
     embedding table never the top isolated priority
     module ordering family-dependent
     joint damage is not the sum of parts at INT3

The Increment

One sentence: Before this paper, PTQ for embedders reused LLM heuristics; after it, a measured map shows those heuristics do not transfer as stated, and a 109M INT3 student dominates its 0.6B teacher on size and quality — but only inside the distillation task.

Core Mechanism

This is a measurement paper, so the mechanism is the experimental design. The authors quantize five checkpoints from four architecture families across a grid of bit widths and group sizes, and isolate the embedding, attention and feed-forward blocks at each width. That isolation is what turns anecdotes into a map.

Three results carry the weight. The embedding table never emerges as the dominant isolated protection priority in any family, despite being the largest tensor in several of them. Module sensitivity does not survive as a transferable ordering: at INT4/g16 the spread between modules is too small to allocate against; at INT3 the ordering becomes family-dependent and joint damage stops being the sum of its parts; at INT2, comparable reconstruction error accompanies retention ranging from 1.3 to 65.9 percent of full precision. A cheap reconstruction proxy helps screen uniform bit widths but is substantially less reliable for choosing which tensors to protect — its apparent grid-wide strength is a range-extension artifact.

The constructive arm is a distilled 109M student at INT3 holding 78.04 NDCG@10 in 68.4 MB, which dominates the extreme-PTQ arm of its own 0.6B teacher, 297.9 MB at 64.46, on both size and quality — but only inside the task it was distilled for. Sizes are byte counts of files that exist, not arithmetic estimates.

   WHAT THE MAP ACTUALLY SHOWS

   uniform PTQ grid
     bit width x group size
     modules isolated per width
         |
         +-- embedding table: NOT the top priority
         |
         +-- module ordering: FAMILY-DEPENDENT
         |     INT4/g16: spread too small to allocate
         |     INT3: joint damage != sum of parts
         |     INT2: same recon error, retention 1.3% .. 65.9%
         |
         +-- recon proxy: OK for width screening
         |     WEAK for choosing which tensors to protect
         |
         v
   distilled 109M @ INT3
     78.04 NDCG@10 in 68.4 MB
     beats 0.6B teacher PTQ (64.46 / 297.9 MB)
     ONLY on the distillation task

Think of it as shrinking a filing system with a photocopier checklist designed for a different building. The checklist says “the index drawer is sacred” and “score cabinets by how dusty they look.” On the new building, the index drawer is not what breaks service; dust scores do not order cabinets across floors; and two cabinets that each look fine alone fail together when the elevator is smaller. The paper is the building survey that makes the checklist’s assumptions visible — and shows a purpose-built annex (the distilled student) only serves its own department.

Key Concepts

• Heuristic transfer failure: rules that work for generative LMs need not order modules the same way in embedders.
• Non-additive damage at low bits: protecting modules independently can miss joint failure at INT3.
• Task-bounded distillation win: a small student can beat a PTQ’d teacher only on the distribution it was distilled for.

Framework Shift

Before (borrowed LLM advice):         After (measured embedder map):
  protect embedding table first       table is not top isolated priority
  bits by module sensitivity          ordering is family- and width-dependent
  recon proxy as allocation guide     proxy screens widths, not protection sets
  teacher PTQ as default shrink       distilled INT3 student wins in-task only

From reusing LLM quantization folklore on retrieval models, to measuring where each rule breaks by architecture and bit width, the core shift is from advice to a map.

Expert Assessment

Problem choice: Excellent practical gap. Retrieval stacks are full of embedders being PTQ’d with rules that were never validated on ranking losses.

Method maturity: The isolation grid is the right instrument. An unexamined assumption worth naming: that NDCG@10 retention is the production objective that matters — latency, embedding dimension stability, and cross-encoder rerank coupling are out of scope, and a “safe” module choice under NDCG may still break a downstream reranker.

Experimental integrity: Best part of the paper: reporting non-transfer rather than inventing a new universal rule, and insisting sizes are real file bytes. The 1.3–65.9% retention band at matched reconstruction error is the kind of uncomfortable number that prevents bad defaults.

Writing quality: Dense but honest. The student-vs-teacher comparison would land harder with a second, held-out retrieval task showing where the student falls over.

Verdict: weak accept — a useful production map and a negative result about folklore; contribution is measurement, not a new quantizer.

Takeaways

• Do not allocate PTQ protection bits for embedders using LLM module-sensitivity folklore without re-measuring on your architecture.
• At INT3, audit joint module damage; independent per-module scores can lie.
• If you need aggressive shrink, consider task-specific distillation — but treat the student as task-bound, not a drop-in general embedder.