CODEC: Tracing Causal Paths Through Neural Networks by Decomposing Contributions, Not Just Activations

Paper: 2603.06557 Authors: Joshua Brendan Melander, Zaki Alaoui, Shenghua Liu, Surya Ganguli, Stephen A. Baccus Categories: cs.LG, q-bio.NC

The Gap

Interpretability research has been stuck analyzing what neurons represent (their activation patterns) rather than what they do (their causal effects). Methods like feature visualization, probing classifiers, and activation atlases all ask: “What concept does this neuron encode?” They look for correlations between hidden-layer activations and human-interpretable features. But correlation isn’t causation. A neuron might activate for “dog ears” yet contribute negatively to the “dog” class prediction, or vice versa. Activation-based methods can’t see this because they ignore the downstream computational graph.

The gap: we lack tools to trace causal influence through the network’s nonlinear transformations. Existing gradient-based attribution methods (saliency maps, integrated gradients) attribute to inputs, not to intermediate computations. Circuit analysis manually traces connections but doesn’t scale. We need a systematic way to decompose how hidden neurons causally contribute to outputs across layers.

Problem: Activations != Causal Effects
   |
   v
Assumption: Contributions (activation * downstream weight)
            can be sparsely decomposed into interpretable motifs
   |
   v
Method: Train sparse autoencoders on contribution vectors
        to find recurring causal patterns
   |
   v
Evidence: Contributions grow sparser across layers,
          positive/negative effects decorrelate,
          sparse modes enable targeted interventions
   |
   v
Conclusion: Contribution decomposition reveals causal
            structure invisible to activation analysis

The Increment

One sentence: Before, we asked “what does this neuron detect?” — now we can ask “how does this neuron causally push the output, and through which sparse motifs?”

Core Mechanism

CODEC operates in three stages. First, it computes contribution vectors for each hidden neuron: the neuron’s activation multiplied by its outgoing weights to the next layer. This captures how much each neuron pushes or pulls on downstream computations. Unlike raw activations, contributions are signed (positive or negative) and dimensionally aligned with the next layer’s neurons.

Second, CODEC trains a sparse autoencoder on these contribution vectors across many inputs. The autoencoder learns to reconstruct each contribution vector as a sparse linear combination of learned basis vectors (called “contribution modes”). The sparsity constraint forces the autoencoder to discover recurring patterns — motifs of how neurons jointly influence outputs. Each mode represents a stereotyped causal pathway: a specific way that hidden neurons combine to drive particular downstream effects.

Third, CODEC uses these learned modes for interpretation and intervention. By examining which input features activate each mode, researchers can visualize what image components trigger specific causal pathways. By ablating or amplifying modes, they can surgically manipulate network behavior at the level of causal mechanisms rather than individual neurons.

Input Image
    |
    v
[Hidden Layer Activations]
    |
    | (multiply by outgoing weights)
    v
[Contribution Vectors] <--- what each neuron does to next layer
    |
    | (sparse autoencoder)
    v
[Sparse Modes] <--- recurring causal motifs
    |
    +---> Interpretation: visualize what triggers each mode
    |
    +---> Intervention: ablate/amplify modes to control output

Think of CODEC like analyzing a company’s financial flows instead of just employee headcounts. Activation analysis is like counting how many people work in each department — you see who’s present but not what they’re doing. CODEC tracks the money: it follows how each department’s budget (activation) gets allocated to downstream projects (weighted contributions). The sparse autoencoder is like an accountant finding recurring spending patterns — “R&D always funds these three initiatives together” or “Marketing’s budget splits into brand vs. performance channels.” These patterns (modes) reveal the company’s actual operational structure, not just its org chart. When you want to change outcomes, you intervene on spending patterns (modes), not individual employees (neurons).

Key Concepts

• Contribution vs. Activation: An activation tells you a neuron fired strongly. A contribution tells you what that firing did to the output. Concretely: if neuron i has activation a_i and connects to neuron j with weight w_ij, its contribution to j is a_i × w_ij. This is signed (can be negative) and directional (points toward specific downstream neurons). A neuron can activate strongly but contribute weakly if its weights are small, or contribute negatively even when active if weights are negative. Contributions capture causal influence; activations don’t.

• Sparse Autoencoder for Decomposition: You have 10,000 contribution vectors (one per neuron per input). Each vector is high-dimensional (size of next layer). A sparse autoencoder learns, say, 100 basis vectors (modes) such that any contribution vector can be approximated as a sparse sum of ~5-10 modes. The sparsity constraint (L1 penalty on coefficients) forces the model to reuse a small vocabulary of patterns rather than memorizing each vector. This is like PCA but with sparsity: you get interpretable, reusable motifs instead of dense, entangled components.

• Decorrelation of Positive/Negative Effects: CODEC found that in early layers, neurons that increase class A’s score often simultaneously decrease class B’s score (correlated effects). In deeper layers, these effects separate: some modes purely boost A, others purely suppress B (decorrelated). This wasn’t predicted. It suggests the network progressively disentangles “evidence for X” from “evidence against Y” as computation proceeds — a form of causal structure that activation analysis misses entirely because it can’t see the sign of influence.

Framework Shift

Before (activation analysis):        After (CODEC):

Input                                Input
  |                                    |
  v                                    v
[Neurons: what fires?]               [Neurons: activations]
  |                                    |
  | (correlation)                      | (multiply weights)
  v                                    v
"This neuron detects dogs"           [Contributions: causal push/pull]
                                       |
                                       | (sparse decomposition)
                                       v
                                     [Modes: recurring causal motifs]
                                       |
                                       v
                                     "This mode suppresses cat 
                                      while boosting dog via 
                                      texture pathway"

From “what concepts do neurons encode?” to “what causal operations do computation pathways perform?” — the core shift is from representational correlation to mechanistic decomposition.

Expert Assessment

Problem choice: This is a real gap. The field has been circling around causality for years (circuit analysis, causal scrubbing, path patching) but lacked a scalable, layer-agnostic method. CODEC sits at the intersection of mechanistic interpretability and sparse coding, which is timely given recent interest in SAEs for language models. Not groundbreaking conceptually — contribution analysis is old, SAEs are standard — but the combination addresses a genuine need.

Method maturity: Clever but not revolutionary. The core insight (decompose contributions, not activations) is simple and should’ve been tried earlier. The use of SAEs is competent but unexciting — they’re the default tool for sparse decomposition now. I’d want to see ablations: what if you used NMF instead? What if you decomposed gradients rather than contributions? The paper doesn’t explore alternatives, which makes me wonder if SAEs are necessary or just convenient.

Experimental integrity: Solid but narrow. The image classification experiments (ResNet, VGG) are convincing for the core claims about sparsity and decorrelation. The retina model application is interesting but feels tacked on — it’s a different domain with different evaluation criteria, and the paper doesn’t deeply engage with neuroscience literature. Baselines are reasonable (comparing to activation-based methods) but not exhaustive (no comparison to other causal attribution methods like path patching). The decorrelation finding is intriguing but under-analyzed: why does this happen? Is it universal or architecture-specific?

Writing quality: The paper front-loads motivation well but buries key details. The sparse autoencoder training procedure is vague (hyperparameters, convergence criteria, how many modes to use). The visualization section is pretty but doesn’t rigorously validate that modes are more interpretable than raw neurons — it’s mostly qualitative. The retina section feels rushed, like they wanted a second application but didn’t have space to do it justice. If I were reviewing, I’d ask them to either expand the retina analysis or cut it and deepen the image classification results.

Verdict: weak accept — addresses a real gap with a straightforward method, but the execution is workmanlike rather than insightful, and the paper doesn’t fully explore the implications of its own findings.

Takeaways

The big steal: when analyzing any neural network, compute contributions (activation × outgoing weights) instead of just activations. This is trivial to implement and immediately gives you signed, causal information. You can do this without CODEC’s full machinery.

The sparse decomposition trick: if you have high-dimensional vectors you want to interpret (contributions, gradients, embeddings), train a sparse autoencoder to find recurring patterns. This is more interpretable than PCA and scales better than manual clustering. Useful beyond interpretability — think data compression, anomaly detection, feature engineering.

The decorrelation observation: if you’re designing architectures or loss functions, consider whether you want positive/negative effects to decorrelate across layers. CODEC suggests this happens naturally in classification networks, but you could explicitly encourage or discourage it depending on your task. For instance, in multi-label classification, you might want effects to stay correlated; in disentangled representation learning, you’d want decorrelation.

The intervention angle: sparse modes give you better control knobs than individual neurons. If you’re doing model editing, steering, or adversarial robustness work, decomposing into sparse causal pathways might give you more precise, less brittle interventions. This is underexplored in the paper but seems like the most practically useful direction.