Tracking Equivalent Mechanistic Interpretations Across Neural Networks

Paper: 2603.30002 Authors: Alan Sun, Mariya Toneva Categories: cs.LG, cs.CL

The Gap

Mechanistic interpretability (MI) tries to reverse-engineer neural networks by finding the algorithmic process behind their decisions. Researchers manually discover “circuits” (subgraphs of neurons) that implement specific computations. But there’s a scaling crisis: every new model requires starting from scratch, and there’s no formal way to know if two models are doing “the same thing” algorithmically.

Prior work focuses on finding interpretations for individual models. Circuits are identified through manual exploration or automated methods, but these are model-specific. When you train a new model or change architectures, you can’t reuse previous interpretations. There’s no principled answer to: “Do these two models solve the task the same way?”

Problem: MI doesn't scale or generalize
    |
    v
Assumption: If two models share an interpretation,
            their implementations should be equivalent
    |
    v
Method: Define interpretive equivalence via
        implementation equivalence + estimate it
        using representation similarity
    |
    v
Evidence: Tested on Transformers doing modular
          arithmetic and IOI tasks
    |
    v
Conclusion: Can detect equivalent interpretations
            without explicit description

The Increment

One sentence: Before this paper, you had to manually rediscover interpretations for each model; after, you can automatically detect when different models share the same algorithmic strategy.

Core Mechanism

The method has three layers. First, define what it means for two interpretations to be equivalent: they’re equivalent if every possible way to implement interpretation A is also a valid implementation of interpretation B, and vice versa. This is a set-theoretic definition—two interpretations are the same if their implementation sets are identical.

Second, connect this abstract definition to something measurable. The key insight: if two models have equivalent interpretations, then any circuit implementing that interpretation in model 1 should have a corresponding circuit in model 2 that does the same computation. This means their internal representations must align in a specific way—not just similar, but functionally interchangeable.

Third, estimate this alignment using representation similarity metrics. The algorithm compares how models represent information at different layers. If representations are sufficiently similar (measured by CKA or linear probing), and if swapping components between models preserves behavior, then the models likely share an interpretation.

Model A          Model B
   |                |
   v                v
[Layer 1]  <--->  [Layer 1]  (compare representations)
   |                |
   v                v
[Layer 2]  <--->  [Layer 2]  (check alignment)
   |                |
   v                v
Output A         Output B
   |                |
   +-------+--------+
           |
           v
    Same behavior? --> Equivalent interpretation

Think of it like comparing two chefs. You don’t need to watch them cook to know if they’re following the same recipe. Instead, check if their intermediate prep work looks the same—if chef A’s diced onions could be swapped into chef B’s pan without changing the final dish, they’re probably using the same technique. The “diced onions” are the internal representations, and “swappable” means the representations are functionally equivalent. If at every stage of cooking, you can swap ingredients between kitchens and still get the same meal, the chefs are following the same recipe—even if you never read the recipe itself.

Key Concepts

• Interpretive Equivalence: Two models have equivalent interpretations if they solve a task using the same algorithmic strategy, even if their weights differ. Formally, this means the set of all possible circuit implementations for interpretation A equals the set for interpretation B. It’s not about surface similarity—two models can have completely different architectures but still be interpretively equivalent if they’re computing the same function in the same way. Example: A Transformer and an RNN might both implement “copy the first token” by storing it in a hidden state, making them interpretively equivalent for that subtask despite architectural differences.

• Implementation Set: For a given interpretation (like “the model does modular addition by computing remainders”), the implementation set is all possible circuits that could realize that interpretation. A circuit is a subgraph of the network—specific neurons and connections. The implementation set is huge and implicit; you never enumerate it. But if two interpretations have identical implementation sets, they’re the same interpretation. This shifts the problem from “describe the algorithm” to “compare implementation sets,” which can be done indirectly through representations.

• Representation Similarity as Proxy: The paper’s core technical move is proving that if two models have aligned representations (measured by CKA or linear probing accuracy), and if their behaviors match, then their interpretations are likely equivalent. This works because representations encode what the model “knows” at each layer. If model A’s layer 3 representations can be linearly transformed into model B’s layer 3 representations, and both models make the same predictions, then they’re processing information the same way. It’s a necessary condition: equivalent interpretations must have similar representations. The paper provides sufficient conditions too, involving intervention experiments.

Framework Shift

Before (mainstream approach):        After (this paper):

Model --> Manual exploration         Model A ----+
          |                                      |
          v                                      v
      Find circuits                      Compare representations
          |                                      ^
          v                                      |
      Describe algorithm               Model B --+
                                                 |
(Repeat for each new model)                     v
                                        Equivalent? (yes/no)
                                        
(No reuse across models)             (Reuse interpretations automatically)

From “interpret each model individually” to “detect shared interpretations across models,” the core shift is treating interpretations as equivalence classes rather than descriptions.

Expert Assessment

Problem choice: This is a real gap. MI’s inability to scale is a known bottleneck, and the lack of formal definitions makes it hard to evaluate claims. The problem sits at a critical juncture—MI is moving from toy examples to real models, and needs rigorous foundations. Not manufactured.

Method maturity: Clever insight, not brute force. The move from “describe interpretations” to “compare implementation sets” is elegant. However, the reliance on representation similarity metrics (CKA, linear probing) is a bit hand-wavy—these metrics have known failure modes. The paper acknowledges this but doesn’t fully address when the method breaks. A simpler approach might be direct behavioral testing (ablations, interventions), but that’s computationally expensive. The theoretical framework is solid; the estimation algorithm is pragmatic but imperfect.

Experimental integrity: Baselines are fair—they compare against random models and models with different training seeds. The modular arithmetic and IOI tasks are standard MI benchmarks. Numbers look reasonable, but the experiments are limited to Transformers on small tasks. The big question: does this scale to LLMs? The paper doesn’t claim it does, which is honest. One red flag: the sufficient conditions for equivalence (Theorem 3.2) require strong assumptions about representation alignment that may not hold in practice.

Writing quality: The theory section (Section 3) is dense and could use more intuition before diving into formalism. The case studies (Section 4) are clear but feel rushed—more ablations would strengthen claims. The related work section undersells how this connects to circuit discovery methods. If Section 3 were rewritten with more examples and fewer symbols upfront, the paper would be much more accessible.

Verdict: weak accept — Solid theoretical contribution with practical limitations; needs more empirical validation but opens an important research direction.

Takeaways

Practitioners can steal the core idea: use representation similarity as a cheap proxy for algorithmic equivalence. If you’re fine-tuning a model and want to know if it’s still using the same strategy, compare representations before and after. If they’re aligned, the interpretation likely hasn’t changed. This is faster than re-running full circuit discovery.

The theoretical framework is also useful for evaluation: if you claim two models do the same thing, you now have a formal criterion to test that claim. This could standardize MI benchmarks.

One concrete technique: the paper’s intervention experiments (swapping components between models) are a practical way to test if representations are functionally equivalent, not just statistically similar. This is more robust than CKA alone.

For researchers: the “implementation set” framing reframes interpretation discovery as a search problem over circuits, which could enable automated methods. Instead of manually describing algorithms, search for circuits whose implementation sets match a target interpretation.