Causality is Key for Interpretability Claims to Generalise

Paper: 2602.16698 Authors: Shruti Joshi, Aaron Mueller, David Klindt, Wieland Brendel, Patrik Reizinger, Dhanya Sridhar Categories: cs.LG

Abstract

This paper addresses a critical gap in LLM interpretability research: the tendency to make causal claims that exceed what the evidence can support. The authors argue that many interpretability findings fail to generalize because researchers conflate correlation with causation and make counterfactual claims without proper justification. By grounding interpretability in Pearl’s causal hierarchy and causal representation learning (CRL), they propose a diagnostic framework that matches interpretability methods to the strength of claims they can legitimately support—ensuring findings are both rigorous and generalizable.

Key Contributions

• Causal hierarchy for interpretability: Maps Pearl’s three-level causal hierarchy (association, intervention, counterfactual) to common interpretability practices, clarifying what each level of analysis can justify
• Critique of current methods: Demonstrates how popular techniques like activation patching and ablation studies support only interventional claims, not the counterfactual interpretations often attributed to them
• CRL integration: Shows how causal representation learning provides formal conditions under which high-level variables can be recovered from neural activations
• Diagnostic framework: Offers practitioners a systematic approach to select appropriate methods and evaluations that align claims with available evidence

The Causal Hierarchy in Interpretability

The paper leverages Pearl’s causal ladder to categorize interpretability research:

Level 1: Association - Observational studies that identify correlations between model activations and behaviors. Examples include probing classifiers and representation similarity analysis. These methods answer “what is?” questions but cannot establish causation.

Level 2: Intervention - Experimental manipulations like ablations, activation patching, or steering vectors. These answer “what if we do?” questions, showing how changes to model components affect outputs across a distribution of inputs. Crucially, interventional claims are about average effects over datasets, not specific counterfactuals.

Level 3: Counterfactual - The strongest claims, answering “what would have happened?” for specific instances under unobserved conditions. The authors argue these require either controlled supervision during training or strong untestable assumptions—making most counterfactual interpretability claims unverifiable.

Why Current Interpretability Falls Short

Many interpretability studies implicitly make counterfactual claims when their methods only support interventional ones. For example:

• Activation patching shows that replacing activations from prompt A with those from prompt B changes the output. This is interventional evidence about the effect of that swap.
• However, researchers often interpret this as “this activation encodes concept X for this specific prompt”—a counterfactual claim about what would happen under a hypothetical intervention that wasn’t actually performed.

The distinction matters because interventional findings may not generalize to out-of-distribution prompts or different model states. Without causal grounding, we risk building interpretability tools that work on benchmarks but fail in deployment.

Causal Representation Learning as Foundation

CRL provides the theoretical machinery to formalize when and how high-level causal variables can be identified from low-level representations. The key insight: under certain assumptions (like sufficient variability in the data or known intervention targets), we can provably recover latent causal structures from observations.

For interpretability, this means:

• Specifying which semantic concepts are theoretically recoverable from activations
• Identifying what data or assumptions are needed for valid recovery
• Distinguishing between identifiable causal variables and mere correlates

The authors advocate for interpretability research to explicitly state its position in this framework, making assumptions transparent and claims falsifiable.

Practical Implications

The diagnostic framework guides researchers to:

1. Match methods to claims: Use observational methods for exploratory analysis, interventional methods for causal effect estimation, and recognize when counterfactual claims require additional evidence
2. Evaluate generalization explicitly: Test whether findings hold across different prompts, model checkpoints, and task variations
3. State assumptions clearly: Make explicit what causal assumptions underlie interpretability claims
4. Design better benchmarks: Create evaluation protocols that test whether discovered mechanisms generalize beyond the training distribution

Takeaways

1. Most current interpretability methods support interventional claims at best—counterfactual interpretations require much stronger evidence than typically provided
2. Pearl’s causal hierarchy offers a principled framework for categorizing interpretability research and matching claims to evidence
3. Causal representation learning formalizes when high-level concepts can be reliably extracted from neural activations
4. Generalization failures in interpretability often stem from treating interventional findings as counterfactual truths
5. The field needs standardized practices for stating assumptions, testing generalization, and aligning method capabilities with research claims