Surrogate Fidelity: When Open Models Can (and Can't) Explain Closed Ones

Paper: 2606.32008 Authors: Philippe Chlenski, Zachariah Carmichael, Ayush Warikoo, Chia-Tse Shao, Yingxiao Ye, Aobo Yang, Vivek Miglani, Nehal Bandi Categories: cs.LG

The Gap

Mechanistic interpretability (MI) has made stunning progress — but only on models you can crack open and inspect neuron by neuron. The problem: the models people actually use (GPT, Gemini, Claude) are locked behind APIs. So the field has quietly adopted a working assumption: study a transparent open model, and the insights should transfer to the opaque production model you care about.

Nobody has rigorously tested this assumption. Prior work either studies a single model family, focuses only on prediction-level agreement, or conflates “these models behave similarly” with “these models work the same way internally.” This paper draws a sharp line between three levels of fidelity — prediction, attribution, and representation — and shows they come apart badly.

Mechanistic interpretability wants to explain *how* models think
     |
     v
But production models (GPT, Gemini) are API-only: no weights, no activations
     |
     v
ASSUMPTION: study open models (Llama, Qwen) -> insights transfer to closed ones
     |
     v
UNTESTED: does agreement on *what* (prediction) mean agreement on *why* (attribution)?
     |
     v
THIS PAPER: test fidelity at 3 levels across 11 models x 4 families
     |
     v
FINDING: prediction fidelity drastically overstates attribution fidelity
         white-box signals stable but causally weak
         black-box ablations capture causal structure by design
     |
     v
CONCLUSION: mechanistic insight does NOT automatically transfer
            prediction agreement is insufficient to warrant transfer

The Increment

One sentence: Before this paper, researchers implicitly treated open-model explanations as portable to closed targets; after this paper, we have rigorous evidence that this transfer fails at the causal level, and we know which signals to trust (and which to distrust).

Core Mechanism

The paper’s contribution is a systematic evaluation framework, not a new model or training method. The authors test surrogate fidelity at three distinct levels across eleven models spanning four families (Llama, Qwen, GPT, Gemini).

Prediction fidelity asks: do two models produce the same outputs? This is the easiest test — just compare answers. Attribution fidelity asks: do two models arrive at those answers for the same reasons? This is harder — you need to measure which input features matter, then compare. Representation fidelity asks: do the models have similar internal geometry? This is hardest — you need to align and compare hidden-state spaces.

For API-only models (GPT, Gemini), you can’t access attention heads or activations. So the authors use log-odds as a scalar proxy for the model’s representation space — the log-ratio of probabilities for two classes gives you a single number that encodes how the model “sees” a problem. For attributions on closed models, they use leave-one-out input ablations: systematically remove tokens and measure how predictions change. This is black-box by design — it works through the API.

For open models, they compute the full suite of white-box signals: attention patterns, perturbation magnitudes, gradient-based attributions. Then they compare: does a white-box signal measured on an open model predict the black-box attribution pattern of a closed model?

              Open Model                    Closed Model
           (white-box access)              (API-only access)
                  |                              |
    +------+------+                    +---------+---------+
    |      |      |                    |                   |
 Attn   Perturb  Activations       Log-probs            Ablations
patterns magnitudes                 (log-odds)        (leave-one-out)
    |      |      |                    |                   |
    v      v      v                    v                   v
  White-box signals               Prediction readout   Attribution readout
  (detailed, internal)            (scalar, external)   (behavioral, external)
    |                                   |                   |
    +---------> Compare <--------------+-------------------+
                at 3 levels:
            Prediction   Attribution   Representation
                |              |              |
                v              v              v
            High fidelity  Low fidelity   Moderate
            (models agree  (models disagree (some shared
             on answers)    on reasons)     structure)

Here’s the structural metaphor that makes this click. Imagine two hospital emergency rooms — one with glass walls where you can watch every doctor’s decision, and one behind a closed door where you only see the patient outcome.

Prediction fidelity = do both ERs discharge the same diagnosis? Yes, mostly — both “rooms” say “appendicitis.”

Attribution fidelity = did the doctors reach that diagnosis the same way? The glass-wall doctor ordered a CT scan; the hidden-door doctor relied on blood work alone. Same answer, totally different reasoning. You *cannot walk into the glass ER, watch the doctor order a CT, and conclude the hidden doctor also ordered a CT.

The access-validity inversion is the kicker: the signals you can see clearly through the glass wall (how the doctor holds the scalpel, the order of procedures) are *stable — every glass-wall ER does it similarly — but they tell you almost nothing about what the hidden-door doctor is doing. Meanwhile, a crude external test (what happens if you withhold a symptom from the patient intake form?) is noisy but causally grounded — it directly measures decision-making.

The lesson: detailed observational access to one system does not give you reliable causal knowledge of another, even when outcomes agree.

Key Concepts

• Surrogate Fidelity: Think of it as a “can I use Model A to explain Model B?” score. If I study Llama’s internals and conclude “this model works by attending to sentiment words,” surrogate fidelity measures whether that same explanation applies to GPT. It’s not about whether the models give the same answers — it’s about whether the same *mechanism drives those answers. The paper shows these are very different questions with very different answers.

• Access-Validity Inversion: This is the paper’s most surprising finding. You’d naturally assume that richer information (seeing attention weights, neuron activations) gives you better understanding. It turns out that white-box signals measured on open models are *highly consistent across open models (stable) but weakly predictive of causal attributions on closed models (invalid). Meanwhile, black-box input ablations — which are conceptually cruder — capture causal structure by construction, because they directly measure “what happens when I remove this piece?” It’s the difference between watching someone’s muscle movements (detailed but hard to generalize) versus testing what happens when you blindfold them (crude but causally informative).

• Log-odds as API-compatible readout: Most APIs give you probability distributions over next tokens. The log-odds — log(p(class A) / p(class B)) — compresses this into a single scalar that encodes where the model sits in its representation space. Think of it as the model’s “body language” for how confident and in which direction it leans. It’s the richest signal you can get without weight access, and the paper shows it works well for comparing prediction behavior across models.

Framework Shift

Before (mainstream approach):          After (this paper):

 "I studied Llama,                   "I studied Llama, so
  so I understand GPT"                I understand Llama."
       |                                    |
       |                                    |
  Assumes portability               Demands proof of portability
       |                                    |
       |                                    |
  Prediction-level                  Three-level fidelity test:
  agreement = enough                prediction / attribution / representation
       |                                    |
       |                                    |
  White-box signals                 White-box = stable but weak
  assumed gold standard             Black-box ablations = causally valid
       |                                    |
       v                                    v
  MI insight transfers              Insight transfers ONLY with
  automatically                     explicit fidelity evidence

From treating open models as automatic surrogates to demanding explicit fidelity evidence at multiple levels, the core shift is: prediction agreement is necessary but nowhere near sufficient for mechanistic transfer.

Expert Assessment

Problem choice: This is a genuine and important gap. The MI community has been building an entire research program on open models with the implicit hope that insights generalize. Someone needed to test this. The paper sits at a critical juncture — MI is maturing from “cool demos on toy models” to “can we actually use this in production?” and this question blocks that transition.

Method maturity: The framework is clean and pragmatic rather than technically novel. Using log-odds as a scalar readout and leave-one-out ablations for API models is not rocket science — it’s good engineering. The real contribution is the systematic multi-level evaluation across many model families, not any single methodological trick. There are probably more sophisticated alignment-based approaches to representation comparison being overlooked, but the paper’s simplicity is also its strength.

Experimental integrity: The baselines are fair — they compare within-family and across-family, open-to-open and open-to-closed. Eleven models across four families is decent coverage. The key numbers (prediction fidelity ~0.7-0.9, attribution fidelity often near chance) are striking and consistent. One concern: they focus on binary classification tasks, which may not capture the full complexity of open-ended generation. The log-odds readout is inherently limited to settings where you have a clear class comparison.

Writing quality: The paper is unusually well-structured for a mechanistic interpretability paper. The three-level fidelity taxonomy is clearly introduced and consistently applied. Where they cut corners: the representation fidelity analysis feels underexplored compared to prediction and attribution. Section 5 (representation) is noticeably thinner. A deeper dive into *why the access-validity inversion occurs — what structural property of transformer architectures causes white-box signals to decouple from causal roles — would elevate this from “we observe X” to “we understand X.”

Verdict: weak accept — clean evaluation framework addressing a real methodological blind spot, with strong empirical evidence across model families, though the theoretical explanation for *why the inversion happens remains open.

Takeaways

1. Don’t trust prediction-level agreement as evidence for mechanistic transfer. If you’re using Llama to build interpretability tools meant for GPT, prediction overlap is not enough — you need explicit attribution-level testing. This is directly actionable for anyone building “interpretability as a service.”

2. Prefer black-box ablations over white-box signals when explaining closed models. If you have to choose one method for understanding an API-only model, input ablation gives you causally grounded explanations. Attention patterns from a surrogate model look clean but may mislead.

3. The log-odds readout is a practical tool worth stealing. For anyone doing cross-model comparison with API-only access, log(p_classA / p_classB) is a simple, API-compatible scalar that encodes meaningful representation-space information. No fancy infrastructure needed.

4. The fidelity taxonomy (prediction / attribution / representation) is a useful checklist. Even outside LLMs, any time you’re using one system to explain another — a simulator for the real world, a small model for a large one — these three levels give you a structured way to ask “transfer to what, exactly?”