IF4: Adaptive Block-Scaled Data Types for Smarter 4-bit Quantization

Paper: 2603.28765 Authors: Jack Cook, Hyemin S. Lee, Kathryn Le, Junxian Guo, Giovanni Traverso, Anantha P. Chandrakasan, Song Han Categories: cs.CL

The Gap

NVFP4 has become the go-to 4-bit format for quantizing large language models because hardware supports it and it preserves information reasonably well. But there’s a structural flaw: NVFP4’s floating-point representation creates massive quantization errors on near-maximal values within each 16-value block. When your weight is close to the block’s maximum, the quantization grid is too coarse, and you lose precision exactly where it matters most.

Prior work identified this error distribution problem but didn’t fix it. The field was stuck with a single format per model—either FP4 everywhere or INT4 everywhere—even though different blocks of weights have different statistical properties. Some blocks have uniform distributions (INT4 shines), others have exponential tails (FP4 wins). One size fits none.

Problem: NVFP4 error spikes on near-max values
    |
    v
Assumption: Different weight blocks need different formats
    |
    v
Method: Per-block adaptive format selection (IF4)
    |
    v
Evidence: Lower loss in training, higher accuracy in PTQ
    |
    v
Conclusion: Adaptive formats beat fixed formats

The Increment

One sentence: Before this paper, you picked one 4-bit format for the entire model; after, each 16-value block picks its own format based on its data distribution.

Core Mechanism

IF4 operates on blocks of 16 values. For each block, it computes two candidate quantizations: one using FP4, one using INT4. Both share the same E4M3 scale factor. The system measures quantization error for each candidate and picks the winner. The choice is encoded in the scale factor’s sign bit, which NVFP4 leaves unused (floating-point scales are always positive).

Data flow: raw weights → block of 16 → compute FP4 quantization error → compute INT4 quantization error → compare → select format with lower error → encode choice in scale’s sign bit → store quantized values + signed scale. During inference, the sign bit tells the hardware which decoder to use.

The same principle extends to other bit-widths. IF3 chooses between FP3 and INT3. IF6 chooses between E2M3 and E3M2 (two different floating-point formats). The pattern is universal: exploit unused bits in the metadata to store per-block format decisions.

Raw weights (16 values)
    |
    +---> FP4 quantizer ---> error_fp
    |
    +---> INT4 quantizer --> error_int
    |
    v
  Compare errors
    |
    v
  Pick winner --> encode in scale's sign bit
    |
    v
  [quantized values] + [signed scale]

Think of IF4 like a tailor who stocks two types of fabric—silk (FP4) and denim (INT4). When a customer (weight block) walks in, the tailor measures them and picks the fabric that fits better. Silk drapes well on certain body types (exponential distributions), denim works better on others (uniform distributions). The tailor writes the fabric choice on the garment’s tag (sign bit). When you wear the garment later (inference), you check the tag to know how to care for it (which decoder to use). The key insight: the tag was already there, just unused.

Key Concepts

• Block-scaled quantization: Instead of using one scale factor for the entire model, you divide weights into small blocks (typically 16 values) and give each block its own scale. This lets you adapt to local statistics. If one block has values ranging 0.001 to 0.01 and another has 1.0 to 10.0, they get different scales. The block size is a hardware constraint—GPUs process data in chunks, and 16 aligns with warp sizes.

• Format selection overhead: Every adaptive system pays a cost to make decisions. IF4’s cost is computing two quantizations per block and comparing errors. But this happens offline during model quantization, not during inference. At inference time, you just read the sign bit—a single bit check, essentially free. The paper’s MAC unit design shows this adds negligible hardware complexity.

• Error distribution asymmetry: NVFP4’s floating-point grid has exponentially spaced points. Near zero, you get fine resolution. Near the maximum, gaps between representable values grow huge. If your weight is 0.95 ** max and the nearest grid point is 0.85 * max, you eat 10% relative error. INT4’s uniform grid doesn’t have this problem—every value gets equal spacing. The asymmetry means FP4 and INT4 fail on different inputs, making them complementary.

Framework Shift

Before (NVFP4):                    After (IF4):
                                   
Model weights                      Model weights
    |                                  |
    v                                  v
Divide into blocks                 Divide into blocks
    |                                  |
    v                                  v
Apply FP4 to all blocks            For each block:
    |                                  |
    v                                  +-> Test FP4
[FP4][FP4][FP4][FP4]...               +-> Test INT4
                                       +-> Pick better one
Fixed format,                          |
some blocks suffer                     v
large errors                       [FP4][INT4][FP4][INT4]...
                                   
                                   Adaptive format,
                                   each block optimized

From uniform compression to adaptive compression, the core shift is moving the format decision from model-level to block-level.

Expert Assessment

Problem choice: Real gap. The NVFP4 error distribution issue is well-documented, and the field genuinely lacked a hardware-friendly solution. This sits at the intersection of numerical methods and hardware design—a productive place to work. Not a manufactured problem.

Method maturity: Clever insight, not brute force. The sign bit trick is elegant—you get adaptivity for free by repurposing unused metadata. The extension to IF3 and IF6 shows the idea generalizes. However, the paper doesn’t explore learned format selection (could a small neural net predict the best format without computing both?). That’s a missed opportunity for future work.

Experimental integrity: Baselines are fair. They compare against NVFP4, INT4, and FP4 on standard benchmarks (LLaMA, Mistral). The numbers hold up—consistent improvements across multiple model sizes and tasks. One minor flag: the MAC unit evaluation is simulation-based, not silicon. Real hardware might reveal unexpected bottlenecks, but the design is plausible.

Writing quality: Section 3 (method) is crisp. Section 4 (experiments) drowns in tables—Figure 4 could replace three tables. The related work section front-loads too much history; cutting it by half would improve flow. The paper’s core contribution is clear, but the presentation could be tighter.

Verdict: Strong accept — solves a real problem with an elegant, hardware-aware solution that generalizes beyond the initial use case.

Takeaways

The sign bit trick transfers directly to any block-scaled quantization scheme. If your metadata has unused bits, you can encode per-block decisions without storage overhead. This applies beyond quantization—think compression, sparse formats, or mixed-precision training.

The error distribution analysis (FP4 fails on near-max values, INT4 fails on exponential tails) is a reusable diagnostic. When designing numeric formats, plot error vs. input value. If you see asymmetry, you’ve found an opportunity for adaptive methods.

The MAC unit design shows that format adaptivity doesn’t require complex hardware. A multiplexer and two decoders suffice. If you’re building accelerators, this is a low-cost way to boost model quality without changing the compute pipeline.