From Interpretability Methods to Interpretable Models

Paper: 2609.05399 Authors: Julien Colin, Nuria Oliver, Thomas Serre Categories: cs.CV, cs.HC

The Gap

Over the past decade, explainable artificial intelligence (XAI) in computer vision has assembled an enormous technical toolbox: gradient attribution maps (Grad-CAM, Integrated Gradients), feature visualization via activation maximization, concept-based testing (TCAV), and mechanistic circuit analysis.

Yet despite thousands of published papers, the field has fallen into a self-referential trap. Researchers spend virtually all their energy comparing methods against methods: debating whether attribution technique A produces sharper saliency edges than attribution technique B on a frozen ResNet-50.

Meanwhile, the actual core scientific question — are modern neural network architectures (such as Vision Transformers, ConvNeXts, and Vision-Language Foundation Models) becoming more intrinsically interpretable than their predecessors? — has been almost completely neglected. The field behaves like an astronomy department that spends ten years arguing about the lens coatings of five different telescopes, without ever pointing a single telescope at the stars to measure how galaxies evolve.

[CURRENT XAI PARADIGM: METHOD-CENTRIC]
     Method A (Grad-CAM) vs Method B (Integrated Gradients) vs Method C (TCAV)
                 |
                 v (Evaluated endlessly on frozen legacy backbones)
     Outcome: 10,000 papers arguing about visualization tools,
              while real models grow into trillions of unmapped weights!
                 |
                 +-------------------------------+
                                                 v
                                  [THE PROPOSED MODEL-CENTRIC SHIFT]
                       Point mature XAI tools directly at model architectures!
                       Measure: Does shifting from ConvNets to ViTs to VLMs
                       make internal circuits more aligned with human cognition?

The Increment

One sentence: Before this paper, explainable AI was stuck in a meta-debate comparing visualization techniques; after it, Brown University and ELLIS researchers draw a decisive parallel to systems neuroscience, establishing an actionable, model-centric XAI agenda focused on measuring representational alignment and human mental predictability across model generations.

Core Mechanism

The authors argue that the XAI toolbox is now sufficiently mature to cease inventing new post-hoc attribution masks. Instead, the community must redeploy these instruments toward Model-Centric Explainability along two empirical axes:

1. Mechanistic Representational Characterization: Using representational similarity analysis (RSA), sparse autoencoders, and linear probes to systematically track how different model architectures (ResNet vs ViT vs ConvNeXt vs Multimodal Transformers) structure their latent spaces. Do self-attention heads form more human-aligned semantic concepts than convolutional kernels, or do they distribute reasoning across uninterpretable global superposition?
2. Empirical Human Understandability (Mental Model Predictability): Moving beyond expert confirmation bias. Instead of showing an AI engineer a heatmap to see if they nod approvingly, evaluate non-expert end users (such as radiologists, drivers, or loan officers). Can an operator accurately predict when a model will fail after observing its explanations on 20 examples? This is an empirical behavioral property that must be directly measured via psychophysical experiments, not deduced from mathematical axioms.
3. The Systems Neuroscience Analogy: Neuroscience does not exist to debate fMRI versus EEG; it uses both instruments to map cortical representations in biological brains. XAI must treat artificial neural networks as computational organisms to be mapped and understood over evolutionary generations.

   MODEL-CENTRIC XAI FRAMEWORK

   [Mature XAI Toolbox]
   (Attribution, Circuit Probing, Sparse Autoencoders, Concept Vectors)
                     |
                     v
   +-----------------------------------------------------------+
   |                  MODEL-CENTRIC AUDITING                   |
   |                                                           |
   |  [Axis 1: Representational Mapping]                       |
   |  Track feature geometry across ResNet -> ViT -> VLM       |
   |                                                           |
   |  [Axis 2: Human Operator Mental Models]                   |
   |  Psychophysical tests: Can humans anticipate model bugs?  |
   +-----------------------------+-----------------------------+
                                 |
                                 v
   [Result: Scientific Metric of Architectural Interpretability]
   Select model architectures that are fundamentally safer to deploy.

To illustrate this shift, consider a structural metaphor of deep-sea marine biology. For twenty years, oceanographers argued endlessly in conference halls over whether sonar, underwater lidar, or fluoroscopic cameras had the best color contrast. While they argued, strange new abyssal creatures were swimming past uncataloged. A model-centric marine biologist steps onto the boat, packs all three instruments, dives into the trench, and focuses 100% of their attention on dissecting the giant squid to understand its organs and circulatory system.

Key Concepts

• Model-Centric XAI: An empirical paradigm shift that evaluates and benchmarks machine learning architectures on their intrinsic transparency and cognitive alignment, rather than evaluating explainability methods.
• Mental Model Predictability: An objective, human-in-the-loop metric evaluating whether a user can correctly anticipate whether a model will succeed or fail on a novel input after reviewing its explanations.
• Representational Alignment: The degree to which an artificial network’s internal latent activations correspond to natural human perceptual concepts and task invariants.

Framework Shift

Before (Method-Centric Echo Chamber):
Model (Fixed) ---> [Compare Method 1 vs Method 2 vs Method 3] ---> Paper: "Our heatmap is 3% sharper"
(The actual AI model remains an uninspected black box)

After (Model-Centric Scientific Investigation):
Methods (Fixed) -> [Instrument Suite] -> [Audit ResNet vs ViT vs VLM] -> Finding: Architectural Insight!
(Empirically identifies which model designs are safer and more legible to society)

From inventing endless cosmetic heatmap variations to conducting rigorous comparative anatomy across neural network architectures, the core shift is transforming XAI into an observational science.

Expert Assessment

Problem choice: Profound and clarifying. The XAI literature has suffered from severe diminishing returns, publishing thousands of redundant saliency map papers that fail to make deployed models one bit safer.

Method maturity: Grounding the argument in systems neuroscience and psychophysics provides the exact theoretical rigor that post-hoc machine learning explainability has desperately lacked.

Experimental integrity: The paper provides a comprehensive taxonomic survey of the few pioneering papers that have actually compared model architectures, proving that the gap is real and the tools are already ready to use.

Writing quality: Eloquent, provocative, and deeply scholarly. The critique is incisive without being dismissive of foundational work.

Verdict: strong accept — A visionary position paper that will serve as a guiding compass for the next generation of explainable AI research.

Takeaways

• Stop writing papers that introduce minor variants of Grad-CAM or integrated gradients; the existing toolbox is already good enough.
• When choosing an architecture for high-stakes deployment (e.g., healthcare or autonomy), benchmark its mental model predictability across human operators before looking at test-set accuracy.
• Treat artificial neural networks as empirical systems to be dissected and mapped, following the rigor of cognitive neuroscience.