LLMs as Noisy Channels: A Shannon Perspective on Model Capacity and Scaling Laws

Paper: 2605.23901 Authors: Xu Ouyang, Deyi Liu, Yuhang Cai, Jing Liu, Yuan Yang, Chen Zheng, Thomas Hartvigsen, Yiyuan Ma Categories: cs.LG, cs.AI, cs.IT

The Gap

Existing scaling laws (Kaplan et al., Hoffmann et al.) predict that more parameters and more data monotonically improve LLM performance. These power-law formulations work well in the clean regime but break down when you introduce perturbations: catastrophic overtraining (performance collapses after a certain token count), quantization-induced degradation (lower precision hurts more than predicted), and fine-tuning instability. The core issue is that monotonic power laws have no mechanism to model noise or capacity limits—they assume infinite headroom.

Recent perturbation-aware laws (Gadre et al., Muennighoff et al.) add noise terms but still treat scaling as fundamentally monotonic. They patch the symptoms without addressing the root cause: LLM training is not just about accumulating signal, it’s about transmitting signal through a noisy medium. When noise dominates, more capacity or more data makes things worse, not better.

Problem: Scaling laws fail under noise
    |
    v
Assumption: LLM training = information transmission over noisy channel
    |
    v
Method: Map parameters to bandwidth, tokens to signal power
        Apply Shannon-Hartley theorem to derive capacity limit
    |
    v
Evidence: Fit on Pythia/OLMo2 under Gaussian noise, quantization, fine-tuning
          Extrapolate to unseen model sizes and token counts
    |
    v
Conclusion: Shannon Scaling Law captures U-shaped degradation
            Outperforms monotonic and perturbation-aware baselines

The Increment

One sentence: Before this paper, scaling laws assumed more is always better; after this paper, we have a principled capacity limit where scaling without preserving signal-to-noise ratio amplifies noise and degrades performance.

Core Mechanism

The Shannon Scaling Law models LLM training as transmitting information (the learning signal from data) through a noisy channel (the model’s parameter space). Model parameters act as channel bandwidth—more parameters increase capacity. Training tokens act as signal power—more tokens strengthen the signal. But the channel also has intrinsic noise: random initialization, optimization stochasticity, quantization error, distribution shift.

The key insight comes from the Shannon-Hartley theorem, which states that channel capacity grows logarithmically with signal-to-noise ratio (SNR). If you scale bandwidth (parameters) without proportionally scaling signal power (tokens), SNR drops. Below a critical SNR threshold, the channel amplifies noise faster than it transmits signal. This creates a U-shaped loss curve: initial improvement as signal dominates, then degradation as noise takes over.

Mathematically, the authors derive a loss function with three terms: a monotonic power-law term (classical scaling), a noise amplification term (grows with parameters and shrinks with tokens), and a baseline noise floor. The noise term has a sigmoid-like transition controlled by SNR, which captures the shift from signal-dominated to noise-dominated regimes.

Training Data (tokens T)  --->  [Signal Power]
                                      |
                                      v
Model Parameters (N)      --->  [Channel Bandwidth]
                                      |
                                      v
                            +-------------------+
                            |  Noisy Channel    |
                            |  (optimization,   |
                            |   quantization,   |
                            |   distribution)   |
                            +-------------------+
                                      |
                                      v
                            [Transmitted Signal]
                                      |
                                      v
                            Loss = f(SNR, N, T)
                            
SNR = Signal / Noise ~ T / (N * noise_factor)
When SNR high: more N, more T -> lower loss
When SNR low:  more N -> amplifies noise -> higher loss

Think of it like a radio receiver. The model is the antenna—bigger antenna (more parameters) can pick up more signal. The training data is the broadcast power—stronger broadcast (more tokens) means clearer reception. But there’s always static in the air (noise). If you build a huge antenna but don’t increase broadcast power proportionally, the antenna picks up more static than signal. You get worse reception, not better. The Shannon-Hartley theorem tells you exactly when this flip happens: when SNR drops below a critical threshold, scaling the antenna backfires.

Key Concepts

• Signal-to-Noise Ratio (SNR) in LLM Training: In classical information theory, SNR is the ratio of signal power to noise power. Here, signal is the learning gradient from data—the useful information that updates parameters toward better predictions. Noise is everything that corrupts this gradient: random initialization, stochastic optimization, quantization rounding, distribution mismatch between pretraining and fine-tuning. High SNR means the model learns from data. Low SNR means the model learns from noise. Concretely, if you train a 12B model on 10B tokens, SNR is high—each parameter sees enough data to converge. If you train the same 12B model on 1B tokens, SNR is low—parameters overfit to noise because they don’t see enough signal. The Shannon Scaling Law formalizes this: SNR ~ T / (N ** noise_factor), where noise_factor captures perturbation severity (quantization bits, fine-tuning distribution shift, etc.).

• Channel Capacity for LLMs: Shannon’s channel capacity is the maximum rate at which information can be reliably transmitted over a noisy channel. For LLMs, capacity is the maximum amount of knowledge the model can extract from data given its parameter count and noise level. The Shannon-Hartley theorem says capacity = bandwidth ** log(1 + SNR). In LLM terms: effective model capacity = parameter_count * log(1 + tokens / (parameters * noise)). This explains catastrophic overtraining: if you keep adding tokens without adding parameters, SNR grows and capacity increases. But if you add parameters without adding tokens, SNR drops and capacity shrinks. The model has more “space” but less “signal” to fill it, so it fills the space with noise instead. This is why a 12B model trained on 100B tokens can outperform a 12B model trained on 300B tokens under high noise—the latter crossed the capacity threshold.

• U-Shaped Loss Curves: Classical scaling laws predict monotonic improvement: loss decreases as you scale parameters or tokens. But under noise, loss curves become U-shaped: initial improvement, then degradation. The Shannon Scaling Law captures this with a noise amplification term that grows with parameters and shrinks with tokens. At low parameter counts, the power-law term dominates—more parameters mean lower loss. At high parameter counts with insufficient tokens, the noise term dominates—more parameters mean higher loss. The inflection point is where SNR crosses the critical threshold. This explains quantization degradation: lower precision increases noise_factor, which shifts the U-curve left—the inflection point happens at smaller model sizes. It also explains fine-tuning instability: distribution shift increases noise_factor, so models that were undertrained (high SNR) in pretraining become overtrained (low SNR) after fine-tuning.

Framework Shift

Before (monotonic power laws):        After (Shannon Scaling Law):

Loss                                  Loss
  |                                     |
  |  \                                  |  \      /
  |   \                                 |   \    /  <- U-shape
  |    \                                |    \  /
  |     \                               |     \/
  |      \___                           |      \___
  +----------> Scale                    +----------> Scale
  
Always improve with                   Capacity limit:
more parameters/tokens                SNR threshold determines
                                      when scaling backfires
                                      
Assumption:                           Assumption:
Signal accumulates                    Signal competes with noise
indefinitely                          Channel has finite capacity

From monotonic accumulation to capacity-constrained transmission, the core shift is recognizing that LLM training is not just about adding more—it’s about maintaining the balance between signal and noise.

Expert Assessment

Problem choice: This is a real gap. Catastrophic overtraining and quantization degradation are well-documented empirical phenomena that existing scaling laws cannot explain. The problem sits at a critical juncture: as models scale to trillions of parameters and training becomes more expensive, understanding when scaling stops working is not academic—it’s economically vital. The Shannon framing is elegant because it unifies multiple failure modes (overtraining, quantization, fine-tuning instability) under a single theoretical lens.

Method maturity: The core insight—mapping LLM training to Shannon’s noisy channel—is genuinely clever, not brute force. The mathematical derivation is clean and the connection to information theory is well-motivated. However, the noise_factor parameter is somewhat ad-hoc: it’s fitted per perturbation type rather than derived from first principles. A more mature version would predict noise_factor from perturbation characteristics (e.g., quantization bits, distribution divergence metrics) without fitting. The authors acknowledge this limitation but don’t fully address it.

Experimental integrity: Baselines are fair—they compare against Chinchilla, Gadre’s perturbation-aware law, and Muennighoff’s quantization law. The experiments span multiple model families (Pythia, OLMo2), perturbation types (Gaussian noise, quantization, fine-tuning), and tasks (math, QA, code). The extrapolation test is particularly strong: fitting on models up to 6.9B and predicting 12B is a genuine out-of-distribution test. The R^2 scores are impressive (0.847 pooled), but I’d like to see error bars and confidence intervals—right now it’s point estimates only. One red flag: the paper doesn’t test on models larger than 12B, so we don’t know if the law holds at frontier scale (70B+, 405B).

Writing quality: The introduction is excellent—clear motivation, crisp problem statement. The related work section is thorough. The method section is dense but readable. The experiments section is where the authors cut corners: figures are small, axis labels are hard to read, and the ablation studies are buried in the appendix. The discussion section is too short—it raises interesting questions (e.g., can we predict noise_factor from perturbation characteristics?) but doesn’t pursue them. Rewriting the experiments section with larger figures, clearer captions, and inline ablations would elevate the whole paper.

Verdict: Strong accept — This paper makes a genuine theoretical contribution (Shannon framing), solves a real problem (non-monotonic scaling), and validates the theory with solid experiments. The noise_factor parameter is a weakness, but the core insight is strong enough to stand on its own. This will influence how people think about scaling laws going forward.

Takeaways

For practitioners: Before scaling your model, estimate SNR. If you’re training a 10B model, you need at least 100B tokens to stay in the signal-dominated regime (assuming moderate noise). If you’re quantizing to 4-bit, expect the capacity threshold to shift left—a model that was undertrained at 16-bit may be overtrained at 4-bit. If you’re fine-tuning on a narrow domain, treat it as a high-noise regime: use smaller models or more data to preserve SNR.

For researchers: The noise_factor parameter is the next frontier. Can we predict it from perturbation characteristics? For quantization, noise_factor should relate to bit width and rounding error. For fine-tuning, it should relate to distribution divergence (e.g., KL divergence between pretraining and fine-tuning data). Deriving these relationships would make the Shannon Scaling Law fully predictive, not just descriptive.

For theorists: The Shannon framing opens a new line of inquiry: what other information-theoretic tools apply to LLM training? Can we use rate-distortion theory to understand compression-performance tradeoffs? Can we use channel coding theory to design better optimization algorithms? The noisy channel metaphor is not just a scaling law—it’s a lens for rethinking the entire training pipeline.