Compressed Computation is (probably) not Computation in Superposition

Paper: 2606.14673 Authors: Jai Bhagat, Sara Molas-Medina, Giorgi Giglemiani, Stefan Heimersheim Categories: cs.LG

The Gap

The idea of computation in superposition — where a network uses fewer components to compute more functions than the number of components would naively allow — is an active frontier in mechanistic interpretability. Braun et al. (2025) introduced the Compressed Computation (CC) toy model: a single-layer network with 50 neurons that appears to compute 100 ReLU functions, achieving a loss better than expected if it only represented 50 functions. This sparked excitement as a potential minimal example of superposition.

But the CC model has a noisy residual stream — a design detail that mixes input features in a random, fixed way before they reach the neurons. The original authors did not separate the effect of this mixing from genuine superposition. The gap is that the field lacked a careful attribution of the model’s performance: is it real computation in superposition, or just re-encoding a mixture of signals?

This paper shows it is mostly the latter. The authors decompose the loss, analyse the learned weights, and build a baseline that captures the mixing effect — and effectively kill the idea that CC is a superposition example.

Problem: CC model "computes 100 ReLUs with 50 neurons"
         |
         v
Prior assumption: This is computation in superposition
         |
         v
Paper's approach: Separate loss into ReLU term + mixing term
                  Analyse neuron directions via eigendecomposition
                  Build SNMF baseline that only uses mixing matrix
         |
         v
Evidence: Performance scales with mixing matrix magnitude
          Without mixing, loss collapses to expected 50-function floor
          Neuron weights live in top-50 eigendirections of mixing matrix
         |
         v
Conclusion: CC is not computation in superposition

The Increment

One sentence: Before this paper, CC was taken as evidence that superposition can compress computation; after this paper, we know it merely exploits an accidental mixing of labels.

Core Mechanism

The CC model works like this: you have 100 target ReLU functions $\{\text{ReLU}(w_i^T x)\}$. The model has 50 neurons, each outputting a linear combination of these ReLUs via learned weights. But there is a catch: the training labels are not the clean ReLU outputs. Instead, the input passes through a noisy residual stream that linearly mixes the 100 ReLU outputs down to 50 dimensions. Let’s call this mixing matrix $M$ (size 50x100). The model’s actual training target is $y_{\text{mix}} = M \cdot [\text{ReLU outputs}]$, so the model learns to output 50 numbers that approximate this mixed signal — not the original 100 ReLUs.

The authors split the training objective into two parts:

They show that the loss can be decomposed into a term that depends on how well the model reconstructs the original ReLU functions (the “superposition” term), and a term that depends only on the mixing matrix $M$ (the “mixing” term). The mixing term grows with the magnitude of $M$. When the authors remove the mixing (set $M$ to identity with proper padding), the model’s performance drops to the level expected for 50 functions.

Further, the learned neuron weight vectors concentrate in the subspace spanned by the top 50 eigenvalues of $M^T M$. This means the model is essentially learning the eigen-decomposition of the mixing process — not superimposing 100 independent functions.

                          Mixing matrix M (50 x 100)
                               |
100 clean ReLUs ------> 50 mixed signals y_mix
                               |
Input x ---> 50 neurons -----> predictions p
                               |
                     Loss = ||p - y_mix||^2
                         = ||p - M y_clean||^2
                         = "ReLU term" + "mixing term"

Structural metaphor: The Office Coffee Machine

Imagine your office coffee machine has 100 different bean varieties (the 100 ReLU functions). But the machine has a single grinder with a noisy internal pipe that randomly blends beans together before they reach the brew basket. The brew basket can only produce 50 cups at a time (the 50 neurons). Each cup tastes like a mixture of multiple bean varieties.

A colleague proudly claims: “Look! With 50 cups, I can serve 100 distinct coffee flavors — that’s superposition!” But actually, the machine is just serving the blended output of the noisy pipe. The cups aren’t individually representing single bean varieties; they’re each a linear combination determined by the pipe’s mixing pattern.

The key insight: the model’s performance comes from learning that mixing pattern — not from cleverly packing 100 independent signals into 50 neurons. The pipe (mixing matrix) does the heavy lifting.

Key Concepts

• Computation in superposition: A situation where a single neuron (or a small set of neurons) simultaneously represents multiple features, allowing a network to compute more functions than it has dimensions. The classic example is the “toy models of superposition” where sparse features are packed into fewer neurons. In this paper, the authors argue the CC model does not exhibit this phenomenon.

• Mixing matrix $M$: A fixed, random matrix that maps 100 original signals down to 50 mixed signals. It arises from the noisy residual stream in the CC design. The authors show that the model’s performance is proportional to the strength of this matrix — bigger $M$ entries yield better loss. If you eliminate the mixing (set $M$ to a 50x100 matrix that simply selects 50 out of 100 signals), the model cannot beat the baseline of representing only 50 functions. This proves the mixing is the source of the apparent compression.

• Semi-non-negative matrix factorization (SNMF): A simple baseline that factorizes the training targets into a non-negative matrix (representing the 50 neuron activations) times the mixing matrix. It recovers the qualitative loss profile of the trained model, though it doesn’t match the full performance. This baseline requires no learning — it directly solves for the best linear decomposition given the mixing structure. Its closeness to the trained model’s behavior is a smoking gun: the model is doing little more than discovering the same SNMF solution.

Framework Shift

Before (superposition view):              After (mixing view):
100 ReLUs                                 100 ReLUs
   |                                         |
   v                                         v
50 neurons (squeeze)                       Mixing matrix M (50 x 100)
   |                                         |
   v                                         v
Output = 100 ReLUs (reconstructed)         50 mixed signals (training targets)
                                                 |
                                                 v
                                              50 neurons learn to match targets
                                                  |
                                                  v
                                              Output = 50 dimensions (not 100)

One sentence: From expecting superposition to realizing signal mixing — the core shift is recognising that the apparent compression is an artefact of the training target being a linear mixture, not a genuine packing of independent computations.

Expert Assessment

Problem choice: Real gap. The CC model was cited by several interpretability papers as evidence of superposition in action. This paper does the necessary detective work to rule it out. The field needs more of these “negative result” papers.

Method maturity: Clever insight over brute force. The loss decomposition and eigendecomposition are simple and elegant. The SNMF baseline is a nice touch. Could have used a simpler baseline like PCA reconstruction, but SNMF is appropriate given ReLU non-negativity.

Experimental integrity: Baselines are fair — they compare to the expected 50-function floor and to SNMF. The ablation where mixing is removed is compelling. One red flag: the paper only examines a single CC architecture; the conclusion might not generalise to other variants (e.g., deeper layers). But for the specific toy model, the evidence is solid.

Writing quality: The paper is well-structured but dense. The section explaining the loss decomposition could use a more intuitive walkthrough — currently it reads like a math derivation. If the authors rewrote that section with a concrete numerical example (e.g., show a tiny 3x2 mixing matrix), it would dramatically improve accessibility.

Verdict: weak accept — a valuable negative result that debunks a misclaimed phenomenon, though the scope is narrow and the methodology is not revolutionary.

Takeaways

1. Check for label leakage: When a model appears to compute more than its capacity suggests, examine whether the training targets contain unintended linear mixtures. The mixing matrix analysis is a transferable diagnostic.

2. Loss decomposition as a debugging tool: Splitting the loss into interpretable components (ReLU term vs mixing term) revealed the true driver of performance. This technique can be applied to any supervised learning task where labels might be contaminated.

3. Baseline with matching structural assumptions: SNMF provided a no-learning baseline that nearly matched the trained model. This suggests that the model’s optimisation may have discovered a solution close to a simple linear method — a pattern that often indicates the problem is easier than it looks.