Pitfalls in Evaluating Interpretability Agents

Paper: 2603.20101 Authors: Tal Haklay, Nikhil Prakash, Sana Pandey, Antonio Torralba, Aaron Mueller, Jacob Andreas, Tamar Rott Shaham, Yonatan Belinkov Institutions: Technion, Northeastern, MIT, Boston University, Harvard Kempner Institute Categories: cs.AI

Abstract

Automated interpretability systems using LLMs promise to scale neural network analysis beyond human bottlenecks. However, evaluating these systems is challenging: outputs are open-ended, lack ground truth, and may vary across runs. This paper investigates evaluation pitfalls through automated circuit analysis, revealing that replication-based evaluation (comparing to human expert explanations) can mask critical failures including memorization, subjective interpretations, and superficial reasoning. The authors propose an unsupervised intrinsic evaluation based on functional interchangeability as a more robust alternative.

The Problem: Evaluating Automated Interpretability

Why Automated Interpretability?

Interpretability research aims to explain neural network internals:

• How models implement in-context learning
• How they solve arithmetic tasks
• How they track grammatical relationships

Human bottleneck: Formulating hypotheses, designing experiments, iteratively refining conclusions requires expert labor.

Solution: Automated interpretability systems that:

• Execute analysis subroutines autonomously
• Design experiments themselves
• Iteratively refine hypotheses like human researchers

The Evaluation Challenge

Evaluating automated interpretability is inherently difficult:

1. Open-ended outputs: No fixed format or structure
2. No ground truth: Multiple valid explanations exist
3. High variance: Results vary across runs
4. Abstraction levels: Equally valid explanations at different granularities
5. Subjectivity: Human experts may disagree

Key question: How do we know if automated systems produce accurate, trustworthy explanations?

Circuit Analysis: A Case Study

What is Circuit Analysis?

Circuits: Subnetworks of attention heads and MLPs implementing specific computational pathways.

Goal: Understand how models solve tasks by:

1. Identifying circuit components
2. Characterizing functional roles of individual components
3. Grouping components into clusters implementing higher-level mechanisms

Example task: Indirect Object Identification (IOI)

• Input: “When Mary and John went to the store, John gave a drink to”
• Model must predict: “Mary”
• Circuit: Specific attention heads and MLPs that implement this reasoning

The Agentic System

Architecture:

1. Input: Task description, circuit components, example prompts
2. Research agent: Autonomously designs experiments and refines hypotheses
3. Interpretability tools: Logit lens, activation patching, attention patterns
4. Output: Component clusters with natural-language descriptions

Workflow:

Task + Circuit → Research Agent → Experiments → Hypotheses → Clusters
                      ↓
              Interpretability Tools
              (logit lens, patching, attention)

Evaluation Approach

Replication-based evaluation: Compare system outputs to human expert explanations from 6 published circuit analysis tasks:

• IOI (Wang et al., 2022)
• Greater-Than (Hanna et al., 2023)
• Docstring (Tigges et al., 2024)
• And others…

Initial result: System appears competitive with human experts!

The Pitfalls Revealed

Pitfall 1: Subjective and Incomplete Human Explanations

Problem: Human expert explanations are not ground truth.

Evidence:

• Experts may disagree on interpretations
• Explanations may be incomplete or focus on specific aspects
• Different abstraction levels are equally valid

Example: For an attention head, one expert might say “retrieves subject information” while another says “attends to first noun” – both could be correct at different levels of abstraction.

Implication: Agreement with human explanations doesn’t guarantee correctness.

Pitfall 2: Outcome-Based Comparisons Obscure Process

Problem: Comparing final outputs ignores the research process.

What’s hidden:

• How the system arrived at conclusions
• Whether reasoning was sound or superficial
• If experiments were well-designed or arbitrary
• Whether evidence actually supports conclusions

Example: System might produce correct-sounding explanation through:

• Genuine experimental reasoning ✓
• Memorization of published results ✗
• Superficial pattern matching ✗
• Lucky guessing ✗

Implication: Matching outcomes doesn’t validate the research methodology.

Pitfall 3: Memorization and Informed Guessing

Critical finding: Claude Opus 4.1 exhibits evidence of memorization.

Evidence:

• On at least one task, system reproduces published findings suspiciously well
• Can infer task structure from superficial cues (task names, prompt patterns)
• May have seen circuit analysis papers in training data

Experiment: When given only task name and circuit components (no prompts), system can sometimes generate plausible explanations.

Implication: Apparent success may reflect training data contamination rather than genuine interpretability reasoning.

Pitfall 4: Superficial Cue Usage

Problem: LLMs can use shortcuts instead of experimental reasoning.

Examples:

• Task name suggests mechanism (e.g., “Greater-Than” → numerical comparison)
• Prompt structure reveals task type
• Component names hint at function (e.g., “head 15.13” might be known from literature)

Test: Remove informative cues and see if performance degrades.

Finding: Performance often remains surprisingly high, suggesting reliance on superficial patterns.

The Solution: Unsupervised Intrinsic Evaluation

Core Idea

Instead of comparing to human explanations, evaluate based on functional interchangeability.

Principle: If two components have similar functional roles, they should be interchangeable in the circuit.

Methodology

1. Generate component descriptions (automated system output)

2. Compute functional similarity:

• For each pair of components, measure how interchangeable they are
• Use causal interventions: swap components and measure task performance change
• Similar components → small performance change
• Different components → large performance change

3. Compare to description similarity:

• Compute semantic similarity of generated descriptions (using embeddings)
• Check if description similarity correlates with functional similarity

4. Evaluation metric:

Correlation(description_similarity, functional_similarity)

High correlation → descriptions accurately capture functional roles.

Advantages

1. No human labels needed: Fully automated evaluation
2. Objective: Based on measurable task performance
3. Scalable: Can evaluate many components quickly
4. Process-agnostic: Doesn’t matter how system arrived at descriptions
5. Robust to memorization: Functional tests can’t be memorized

Limitations

1. Doesn’t capture all aspects: Functional similarity is one dimension
2. Requires causal interventions: Computationally expensive
3. Task-specific: Interchangeability depends on task
4. Doesn’t validate reasoning process: Still outcome-based

Experimental Findings

Replication-Based Evaluation Results

Apparent success: System matches human expert explanations on most tasks.

Closer examination reveals:

• Some system explanations are more accurate than published human explanations
• Some matches likely due to memorization
• Some successes from superficial cue usage

Intrinsic Evaluation Results

Key finding: Correlation between description similarity and functional similarity varies significantly across tasks.

High correlation tasks: System captures functional roles well Low correlation tasks: Descriptions don’t reflect actual component behavior

Insight: Intrinsic evaluation reveals quality differences masked by replication-based evaluation.

Implications for Automated Interpretability

What We Learned

1. Replication is insufficient: Matching human explanations doesn’t validate automated systems
2. Memorization is real: LLMs may have seen interpretability papers in training
3. Superficial reasoning is common: Systems can succeed without genuine understanding
4. Process matters: Outcome-based evaluation misses critical failures
5. Intrinsic evaluation helps: Functional tests provide more robust signal

Recommendations for Evaluating Interpretability Agents

1. Use multiple evaluation approaches:

• Replication-based (with caveats)
• Intrinsic functional tests
• Process-based evaluation (examine reasoning steps)
• Adversarial tests (remove superficial cues)

2. Check for memorization:

• Test on novel tasks not in literature
• Remove task names and other identifying information
• Use held-out circuits

3. Validate reasoning process:

• Examine experiment designs
• Check if evidence supports conclusions
• Look for coherent research methodology

4. Use unsupervised metrics:

• Functional interchangeability
• Consistency across runs
• Predictive power for downstream tasks

5. Be skeptical of perfect replication:

• If system matches published results too well, investigate memorization
• Look for cases where system disagrees with humans (might be more accurate!)

Broader Implications

For AI Safety

Concern: If we can’t reliably evaluate interpretability systems, we can’t trust them for safety-critical analysis.

Need: Robust evaluation methods before deploying automated interpretability for:

• Detecting deceptive behavior
• Identifying harmful capabilities
• Verifying alignment properties

For Scientific Progress

Challenge: Automated systems might:

• Reproduce existing findings (memorization)
• Generate plausible-sounding but incorrect explanations
• Miss novel insights humans would discover

Opportunity: Well-evaluated systems could:

• Scale interpretability research dramatically
• Discover patterns humans miss
• Enable analysis of larger models

For Methodology

Lesson: Evaluation is as important as the system itself.

Principle: Don’t assume LLM-based systems are reasoning correctly just because outputs look good.

Practice: Invest in evaluation infrastructure alongside system development.

Future Directions

Better Evaluation Methods

1. Causal validation: Test if explanations predict intervention effects
2. Counterfactual reasoning: Check if system understands why alternatives fail
3. Consistency checks: Verify explanations remain stable across perturbations
4. Human-in-the-loop: Combine automated metrics with expert judgment

Improved Systems

1. Explicit reasoning traces: Make research process transparent
2. Uncertainty quantification: Flag low-confidence explanations
3. Adversarial training: Train on tasks designed to prevent shortcuts
4. Modular architectures: Separate hypothesis generation from experimental design

Theoretical Foundations

1. Formalize interpretability goals: What makes an explanation “good”?
2. Develop evaluation theory: Principled frameworks for assessing explanations
3. Study memorization: Understand when and how LLMs memorize interpretability findings

Takeaways

1. Replication-based evaluation is flawed: Matching human explanations doesn’t validate automated systems
2. Memorization is a serious concern: LLMs may reproduce published findings without genuine understanding
3. Superficial reasoning is common: Systems can succeed using shortcuts rather than experimental reasoning
4. Intrinsic evaluation helps: Functional interchangeability provides more robust signal
5. Process matters as much as outcomes: How systems arrive at conclusions is critical
6. Evaluation is hard: No single metric captures all aspects of explanation quality
7. Skepticism is warranted: Don’t trust automated interpretability systems without rigorous evaluation

This work serves as a cautionary tale for the field of automated interpretability. As we build increasingly autonomous systems for analyzing neural networks, we must invest equally in robust evaluation methods. The pitfalls identified here apply broadly to any automated scientific discovery system using LLMs.

The path forward requires:

• Humility: Recognize limitations of current evaluation approaches
• Rigor: Develop multiple complementary evaluation methods
• Transparency: Make reasoning processes auditable
• Skepticism: Question apparent successes and look for failure modes

Only with careful, multi-faceted evaluation can we trust automated interpretability systems to advance our understanding of neural networks safely and reliably.