Why Transformers Have Weird Spikes and Attention Sinks: It's the Architecture, Not the Data

Paper: 2603.05498 Authors: Shangwen Sun, Alfredo Canziani, Yann LeCun, Jiachen Zhu Categories: cs.AI, cs.CL

The Gap

Researchers have noticed two weird things happening in Transformers for years. First, massive activations: a few tokens suddenly spike to extreme values in specific channels, orders of magnitude larger than everything else. Second, attention sinks: certain tokens (often the first token or special tokens) hoover up attention mass from all other tokens, even when they’re semantically irrelevant. Prior work from Dettmers et al. (2022) and Xiao et al. (2023) documented that these phenomena co-occur and often involve the same tokens, but nobody knew why they happen together or what they actually do.

The gap is causal understanding. We observe correlation but lack mechanism. Are these bugs or features? Do they cause each other, or does something else cause both? Without answers, we can’t predict when they’ll appear, can’t design around them, and can’t tell if fixing one breaks the other.

Observation: Spikes + Sinks co-occur in same tokens
      |
      v
Question: Why together? What do they do?
      |
      v
Hypothesis: Pre-norm architecture causes both
      |
      v
Method: Ablate pre-norm, measure decoupling
      |
      v
Evidence: Phenomena separate when pre-norm removed
      |
      v
Conclusion: Architectural artifact with distinct functions

The Increment

One sentence: Before, we knew spikes and sinks co-occur but not why; after, we know pre-norm architecture causes both, and they serve different purposes—spikes as global implicit parameters, sinks as local attention modulators.

Core Mechanism

The paper doesn’t propose a new method—it’s a dissection. The authors systematically ablate architectural components to isolate what causes each phenomenon. They start with standard pre-norm Transformers (LayerNorm before attention/FFN), then switch to post-norm (LayerNorm after), and measure how spikes and sinks behave.

For massive activations, they track activation magnitudes across layers and find that in pre-norm models, certain tokens develop extreme values that persist through the network, creating near-constant hidden states. These act like implicit bias terms—the model learns to encode information in these persistent spikes rather than in the dynamic token representations.

For attention sinks, they analyze attention weight distributions and discover that in pre-norm models, specific tokens accumulate attention mass across heads. But this isn’t about semantic importance—it’s a structural role. Heads use these sink tokens to modulate their outputs: when a head doesn’t need to attend to anything specific, it dumps attention mass into the sink, effectively turning itself off for that position.

Pre-norm Transformer flow:

Token embeddings
      |
      v
[LayerNorm] <--- Pre-norm here
      |
      v
[Attention] ----> Some tokens become sinks
      |           (accumulate attention mass)
      v
[Residual +]
      |
      v
[LayerNorm] <--- Pre-norm here too
      |
      v
[FFN] ---------> Some channels spike
      |           (extreme activations)
      v
[Residual +]
      |
      v
Next layer (spikes persist, sinks continue)

Think of it like a city’s infrastructure. Massive activations are like water towers—they sit at fixed locations (specific tokens/channels) and maintain constant pressure (persistent values) across the entire system (all layers). They’re not part of the dynamic flow; they’re infrastructure that other components rely on. Attention sinks are like traffic roundabouts—they’re designated spots where traffic (attention mass) can be redirected when it doesn’t need to go anywhere specific. A car (attention head) that doesn’t need to reach a particular destination just circles the roundabout (dumps attention into the sink) until it’s needed elsewhere. The pre-norm configuration is the zoning law that requires both water towers and roundabouts to be built in specific locations—change the zoning (switch to post-norm), and suddenly you don’t need either.

Key Concepts

• Massive Activations: Imagine you’re building a neural network to predict the next word. Normally, each token’s representation changes as it flows through layers—“cat” might start as a generic noun, then become “small animal,” then “pet that meows.” But massive activations are different. A few tokens develop extreme values in specific channels (say, channel 247 spikes to 1000 while everything else is around 10) and these values barely change across layers. It’s like the model carved out a few slots to store constant information—not dynamic, context-dependent features, but fixed reference points. Why? Because in pre-norm architectures, the residual connections allow these extreme values to bypass normalization and persist. The model learns to exploit this: instead of encoding everything in the dynamic token representations, it stashes some information in these persistent spikes, effectively using them as extra parameters.

• Attention Sinks: In a normal attention mechanism, each token attends to other tokens based on relevance—“cat” attends strongly to “furry” and “meows.” But attention sinks break this pattern. Certain tokens (often the first token or BOS) attract massive attention from all other tokens, regardless of semantic content. If you look at the attention weights, you’ll see 30-40% of the mass going to the sink token, even when it’s completely irrelevant to the query. This isn’t a bug—it’s a feature. Attention heads use sinks as a “null” option. When a head doesn’t need to attend to anything specific for a particular query, it dumps attention mass into the sink, effectively producing a near-zero output (since the sink token’s value is constant and uninformative). It’s a way for heads to turn themselves off selectively, without having to learn complex gating mechanisms.

• Pre-norm vs Post-norm: This is about where you put LayerNorm in the Transformer block. Pre-norm means you normalize before the attention/FFN operation; post-norm means you normalize after. Sounds trivial, but it changes everything. In pre-norm, the residual connection adds the output directly to the input without normalization, allowing extreme values to accumulate and persist. In post-norm, normalization happens after the residual add, squashing extreme values back to a reasonable range. Pre-norm became popular because it’s easier to train (more stable gradients), but it has this side effect: it creates architectural niches where spikes and sinks can emerge. The model learns to exploit these niches because they’re useful—spikes as implicit parameters, sinks as attention modulators. Switch to post-norm, and these niches disappear, so the phenomena decouple.

Framework Shift

Before (mainstream view):          After (this paper):

Spikes + Sinks = Mystery           Spikes + Sinks = Architecture
                                   
  [Token] ---> [???] ---> Spike      [Token] ---> [Pre-norm] 
     |                                  |              |
     +--------> [???] ---> Sink         |              v
                                        |         [Residual]
Co-occur but why?                       |              |
                                        |              v
                                        +-----> Spike persists
                                        |
                                        +-----> Sink accumulates
                                        
                                   Distinct functions:
                                   Spike = global parameter
                                   Sink = local modulator

From “these are correlated mysteries” to “these are architectural affordances with separate jobs,” the core shift is understanding causality—pre-norm creates the conditions, and the model learns to exploit them for different purposes.

Expert Assessment

Problem choice: This is a real gap, not manufactured. Massive activations and attention sinks have been observed for years and cause practical headaches (quantization breaks, inference optimization fails). Understanding their origin and function is foundational work. It sits at the intersection of interpretability and architecture design—exactly where the field needs clarity.

Method maturity: This is clever forensic work, not brute force. The ablation strategy is simple but effective: change one architectural choice (pre-norm to post-norm) and watch the phenomena decouple. The experiments are clean and the logic is tight. However, the paper could have pushed further—what about other architectural variants (RMSNorm, different residual patterns)? The scope feels slightly narrow.

Experimental integrity: Baselines are fair and the measurements are solid. The authors test across multiple model sizes and datasets, which is good. One concern: the functional claims (spikes as implicit parameters, sinks as attention modulators) are supported by correlation and ablation, but not by direct causal intervention. What happens if you manually inject spikes or remove sinks? The paper stops short of that level of proof.

Writing quality: The paper is well-structured and the main narrative is clear. However, the related work section is thin—it mentions prior observations but doesn’t engage deeply with the mechanistic interpretability literature. The discussion of “why the model learns to exploit these niches” is hand-wavy. A deeper dive into the optimization dynamics would elevate the paper significantly.

Verdict: weak accept — Solid empirical work that clarifies an important mystery, but stops short of full mechanistic understanding and could explore architectural variants more thoroughly.

Takeaways

If you’re building or optimizing Transformers, here’s what to steal:

1. Pre-norm creates architectural niches: If you’re seeing weird outliers or attention patterns, check your normalization placement. Pre-norm is easier to train but creates these side effects. Post-norm avoids them but requires more careful initialization.

2. Spikes are implicit parameters: If you’re doing quantization or pruning, don’t treat all activations equally. Those extreme spikes in a few channels are carrying persistent information across layers—they’re more like model parameters than dynamic activations. Quantize them differently or risk breaking the model.

3. Sinks are attention modulators: If you’re analyzing attention patterns, don’t assume high attention mass means semantic relevance. Sinks are structural—they’re where heads dump attention when they want to produce near-zero output. This matters for interpretability: a head attending to the first token might not be “looking at context,” it might just be turning itself off.

4. Decoupling phenomena requires architectural change: If you want to eliminate spikes or sinks, tweaking hyperparameters won’t cut it. You need to change the architecture (e.g., switch to post-norm). This is a design choice with trade-offs—stability vs side effects.