When Neural Networks Learn to Use Interference: How Feature Correlations Create Semantic Structure

Paper: 2603.09972 Authors: Lucas Prieto, Edward Stevinson, Melih Barsbey, Tolga Birdal, Pedro A. M. Mediano Categories: cs.LG, cs.AI, cs.CV

The Gap

Mechanistic interpretability research has established that neural networks pack more features than dimensions through superposition. The dominant framework, pioneered by Anthropic and others, treats this as a geometric packing problem: sparse, independent features arranged to minimize interference, filtered by ReLUs, forming regular polytopes. This works beautifully in toy settings.

But real data has correlations. Words co-occur. Features activate together. The standard theory says nothing about this — it assumes features are independent and treats all interference as noise to be minimized. Yet language models show semantic clusters and cyclical structures that the polytope story doesn’t predict.

The gap: How does superposition work when features are correlated?

Problem: Real features correlate
   |
   v
Assumption: Standard theory assumes independence
   |                    (treats interference as pure noise)
   v
Method: BOWS - controlled correlated features
   |              (bag-of-words from real text)
   v
Evidence: Interference becomes constructive
   |           (semantic clusters emerge)
   v
Conclusion: Correlations reshape geometry fundamentally

The Increment

One sentence: Before this paper, superposition was understood as minimizing interference between independent features; after, we see that correlated features can arrange themselves to make interference constructive, explaining semantic structure in real models.

Core Mechanism

The authors introduce Bag-of-Words Superposition (BOWS): train a small autoencoder to compress binary bag-of-words vectors from internet text into a lower-dimensional bottleneck. The input is which words appear in a document (not their order or frequency), and the model must reconstruct this from fewer dimensions than there are words.

The key insight: when words co-occur frequently (like “neural” and “network”), the model can place their feature directions close together in the embedding space. When both activate, their interference is constructive — the combined signal is stronger and easier to decode. ReLUs still filter out false positives from inactive features, but now interference between active features helps rather than hurts.

They vary the correlation structure by controlling which text domains to sample from, and they vary the regularization (weight decay) to see how it affects the learned geometry. The result: with realistic correlations and weight decay, features cluster by semantic similarity and form cyclical structures — exactly what’s observed in real language models but unexplained by the standard polytope picture.

Input: Binary word vector (10k dims)
   |
   v
[Encoder: compress to bottleneck]
   |
   v
Bottleneck (512 dims) <-- Superposition happens here
   |                      Features arranged by co-activation
   v
[Decoder: reconstruct]
   |
   v
Output: Reconstructed word vector

Key: Correlated features (e.g., "neural"+"network")
     placed nearby -> interference adds up constructively

Think of it like a crowded subway car. The standard superposition story says: pack people in to minimize contact, everyone stands rigid, any bump is bad. But real commuters who travel together (correlations) can lean on each other — the contact becomes support. The model learns: if these two features usually activate together, place them so their signals reinforce rather than cancel. Weight decay acts like a gentle pressure pushing related features closer, while ReLUs are the doors that keep non-passengers out.

Key Concepts

• Constructive vs Destructive Interference: In the standard view, when two feature directions overlap in the embedding space, their simultaneous activation creates interference that corrupts both signals — this is destructive. But if those features frequently co-activate (like “neural” and “network”), the model can orient them so their combined signal is stronger and more distinct than either alone — this is constructive. The decoder learns to expect this combined pattern. It’s like two radio signals: if they’re random, they create static; if they’re synchronized, they amplify each other.

• Semantic Clustering: When features that co-occur are placed nearby in the embedding space, you get clusters of related concepts. For example, all medical terms might cluster together because they appear in similar documents. This isn’t just dimensionality reduction — it’s the model discovering that interference within a cluster is constructive (these features help each other) while interference across clusters is destructive (these features confuse each other). The geometry encodes the correlation structure of the data.

• Weight Decay as Geometry Shaper: Weight decay penalizes large weights, which pushes the model toward simpler solutions. In superposition, this means: instead of using orthogonal directions for every feature (which requires large decoder weights to separate them), the model prefers to place correlated features close together (smaller weights, since the decoder can use the correlation structure). Weight decay doesn’t just prevent overfitting — it actively shapes the feature geometry to match the data’s correlation structure.

Framework Shift

Before (standard superposition):        After (this paper):

Features = independent                  Features = correlated
    |                                       |
    v                                       v
Minimize all interference              Constructive interference
    |                                   for co-active features
    v                                       |
Regular polytopes                           v
(geometric packing)                    Semantic clusters
                                       (correlation-driven)
    |                                       |
    v                                       v
ReLU filters noise                     ReLU filters false positives
                                       (but preserves constructive signal)

From geometric packing to correlation encoding, the core shift is: interference changes from enemy to ally when features co-activate.

Expert Assessment

Problem choice: This is a real gap. The mechanistic interpretability community has been puzzled by semantic clusters and cyclical structures in language models for a while. The standard polytope story is elegant but clearly incomplete — it’s a spherical cow model. Addressing correlations is the obvious next step, and the authors are right to tackle it.

Method maturity: BOWS is clever and controlled. Using bag-of-words gives you natural correlations from real text while keeping the setup simple enough to analyze. The ablations (varying correlation structure, weight decay) are well-designed. However, the leap from BOWS to “this explains language models” is a bit fast — bag-of-words is still quite different from contextualized embeddings. The mechanism is plausible but needs validation on actual LLM internals.

Experimental integrity: The baselines are fair (comparing to standard sparse coding and the Anthropic toy models). The visualizations are compelling — you can see the clusters form. But I’d want to see more quantitative metrics: how much does constructive interference actually help reconstruction? What’s the trade-off curve between correlation exploitation and false positive rate? The paper leans heavily on qualitative geometry observations.

Writing quality: The introduction is excellent — it clearly motivates the gap. The related work section is thorough. But the results section gets a bit hand-wavy: lots of “we observe” without enough “we measure.” The discussion of cyclical structures feels tacked on — it’s mentioned but not deeply analyzed. Rewriting Section 4 with more quantitative analysis would elevate the whole paper.

Verdict: weak accept — Addresses a real gap with a clever method, but the evidence is more suggestive than conclusive. Needs stronger quantitative validation and clearer connection to real LLMs.

Takeaways

If you’re training sparse autoencoders or doing dictionary learning on neural network activations, consider: your regularization choice shapes the feature geometry based on correlation structure. Weight decay isn’t just preventing overfitting — it’s pushing correlated features together. If your data has strong correlations (like language does), this might be a feature, not a bug.

For interpretability work: look for semantic clusters, not just individual feature directions. If features co-activate, their combined geometry might be more meaningful than either alone. The “one feature = one direction” framing might be too simplistic.

For anyone building compressed representations: interference can be constructive if you know the correlation structure. Instead of fighting to make everything orthogonal, lean into the correlations. Place related features nearby and let them reinforce each other. The decoder can learn to expect these patterns.