When the Data Moves, So Must the Explanation: Rethinking XAI Evaluation

Paper: 2608.06351 Authors: Jerzy Stefanowski Categories: cs.AI

The Gap

The XAI literature has a supply problem in reverse: we have far more explanation methods than ways to tell whether any of them work. Saliency maps (Grad-CAM, integrated gradients), attribution scores (LIME, SHAP), concept-based methods (TCAV), and counterfactual generators (DiCE, Wachter-style optimization) are all standard equipment now. Evaluation, by contrast, mostly happens through functionally-grounded proxies — deletion/insertion curves, faithfulness scores, sparsity, stability under perturbation — all computed without a single human in the loop, and all measured on a frozen test set.

Two boundaries follow from that. First, proxy metrics can disagree with each other and with human usefulness; a saliency map with excellent deletion-curve faithfulness can still be unusable for a radiologist or a satellite-image analyst. Doshi-Velez and Kim’s taxonomy named the human-grounded and application-grounded levels years ago, but the field still overwhelmingly reports level-one numbers. Second — and this is where Stefanowski’s own research program lives — essentially all of this machinery assumes stationarity. Data stream mining has spent two decades on concept drift for predictors (ADWIN, Hoeffding trees, adaptive ensembles), but almost nothing on what happens to explanations when the model underneath them keeps being retrained. A counterfactual like “increase income by 4k and you get the loan” is a promise about a decision boundary; if the boundary moved last Tuesday, the promise is stale and nobody checked.

This paper is a synthesis-and-agenda piece built on three concrete anchors: the DetoxAI system for bias detection and concept unlearning in image classifiers, a human-grounded user study on image-classification explanations, and experiments adapting counterfactual explanations to drifting streams.

PROBLEM: explanation methods multiply; evaluation does not
     |
     v
[ proxy metrics only ] .. faithfulness / sparsity / stability
     |                     no human, no time axis
     v
ASSUMPTION: an explanation is "good" only relative to
            (a) a human user and (b) a live data distribution
     |
     +--> ANCHOR 1: DetoxAI ... bias probe + concept unlearning
     |              question: how do we verify unlearning?
     |
     +--> ANCHOR 2: human-grounded study on image classifiers
     |              question: do users actually get it right?
     |
     +--> ANCHOR 3: counterfactuals over drifting streams
     |              question: is yesterday's recourse still valid?
     |
     v
EVIDENCE: case study + user study + stream experiments
     |
     v
CONCLUSION: evaluate the co-evolving triple
            < data , model , explanation >  over time

The Increment

One sentence: Before, XAI evaluation was a number computed once on a static test set; after, the object of evaluation becomes a *trajectory — data, model, and explanation drifting together — and the paper lays out what breaks when you take that view seriously.

Core Mechanism

There is no single new algorithm here; the mechanism is an evaluation architecture assembled from three parts. The static branch starts from DetoxAI: you have an image classifier that has latched onto a spurious attribute (skin tone, background texture, sensor artifact), you probe for that bias, you apply concept unlearning to surgically remove the model’s reliance on it, and you get a debiased classifier. The hard part is verification. A fairness metric going down tells you the *outputs changed; it does not tell you the model stopped looking at the biased region. So explanation methods get recruited as the measuring instrument — and now you have a circular dependency: you are using an unvalidated tool to validate an unlearning procedure. That circularity is the paper’s opening argument for why evaluation cannot stay proxy-only.

The human branch closes that loop by putting people in it. Instead of asking “does this saliency map score well,” you give participants a classifier’s prediction plus an explanation and measure whether they can predict the model’s behaviour on new inputs, spot when the model is wrong, or identify the spurious feature. This is the human-grounded tier: simplified tasks, non-expert participants, but real behavioural outcomes rather than pixel arithmetic.

The temporal branch is the genuinely under-explored one. Take a stream of instances, a model retrained or incrementally updated as drift is detected, and counterfactual explanations generated per instance. Now several things can rot independently: the counterfactual may become *invalid (applying the suggested change no longer flips the prediction), infeasible (the recommended feature values fall outside the current data distribution), or unstable (near-identical instances get wildly different recourse across time steps). Detecting explanation drift is not the same problem as detecting concept drift — the explanation can shift while the model is frozen, if the input distribution moves, and the model can shift without the explanation changing much. Two clocks, loosely coupled.

      STATIC BRANCH                       TEMPORAL BRANCH
   ......................            ..............................
   [ image dataset ]                 [ stream: x_1 x_2 x_3 ... ]
          |                                      |
          v                                      v
   [ classifier f ]                     [ model f_t (updated) ]
          |                                      |
    +-----+------+                        [ drift detector ]
    |            |                               |
    v            v                        +------+-------+
 [ bias    [ concept                      |              |
   probe ]   unlearning ]              no drift        drift
    |            |                         |              |
    +-----+------+                         v              v
          v                          [ reuse CF ]  [ recompute CF ]
   [ f' debiased ]                         |              |
          |                                +------+-------+
          v                                       v
   [ explanation e ]                     [ e_t : explanation
          |                                 stream over time ]
          v                                       |
   +---------------------------+                  v
   | EVALUATION LAYER          |         [ check: validity?
   | L1 proxy metrics          |           feasibility?
   | L2 human-grounded study   |           stability?
   | L3 application task       |           explanation drift? ]
   +---------------------------+
          |                                       |
          +-------------------+-------------------+
                             v
              track < data , model , explanation >
                    as one co-evolving object

The load-bearing metaphor: a city, its road map, and your GPS directions.

The city is the data distribution. The road map is the trained model — a compressed representation of how the city connects. The turn-by-turn directions the GPS gives you are the explanation, and a counterfactual is exactly a set of directions: “take the next left and you arrive at Approved instead of Rejected.”

Classic XAI evaluation is a cartography contest judged indoors. Two mapmakers submit maps; judges compare them to each other and to a survey taken years ago, and score internal consistency. Nobody drives anywhere. That is the faithfulness metric.

Human-grounded evaluation is handing the map to actual drivers and seeing whether they reach the destination. Sometimes the technically superior map loses, because it labels every alley and the driver can’t find the highway.

Now the temporal branch: the city is under permanent construction. Roads close, new bridges open, one-way directions flip. Three failure modes fall straight out of the metaphor. Invalidity — the left turn you were told to take is now a wall. Infeasibility — the route is technically open but goes the wrong way down a motorway; no human can execute it. Instability — you ask twice, ten minutes apart, and get two completely different routes, so you stop trusting the device. And DetoxAI’s circularity problem becomes: you rerouted around a bad neighbourhood, but the only way to check the new route avoids it is to consult the same map you already suspect is wrong.

Concept drift is the city changing. Explanation drift is the directions changing. They are related but not identical, and the paper’s core claim is that we have monitoring infrastructure for the first and almost none for the second.

Key Concepts

• Concept unlearning: Training doesn’t store facts in labelled boxes; a network’s reliance on “background is snow” is smeared across weights. Concept unlearning tries to identify the *direction in the model’s internal representation that corresponds to an unwanted concept, then suppress the model’s use of it — think of finding the one instrument in an orchestra recording that’s playing off-key and attenuating just that instrument, rather than re-recording the whole piece. It’s cheaper than retraining from scratch on cleaned data, but the “just that instrument” part is exactly what’s hard to verify.

• Human-grounded evaluation: There are three tiers of XAI evaluation. *Functionally-grounded uses a formal proxy — delete the pixels the method called important and see how much accuracy drops. No humans, cheap, runs in CI. Human-grounded uses real people on simplified tasks — show them explanations and check whether they can predict what the model will do next. Application-grounded uses domain experts on their real job, with real consequences. Most published XAI evaluation is tier one because tiers two and three cost money and require ethics approval. The gap between tiers is not a rounding error: methods routinely reverse rank between them.

• Counterfactual validity under drift: A counterfactual is minimal actionable advice — “if this applicant’s debt ratio were 0.31 instead of 0.38, they’d be approved.” It is a claim about where the decision boundary sits *right now. Under drift, the model is retrained and the boundary moves. The advice you gave a customer last month may no longer produce approval even if they follow it exactly. Validity asks: does the recommended change still flip the prediction under the current model? Feasibility asks: is the recommended value still plausible given the current data (a debt ratio of 0.31 may have become rare or impossible)? Stability asks: does similar input keep getting similar advice, or does the recourse jitter every retraining cycle? These are three distinct failure modes and none of them are visible to standard accuracy-under-drift monitoring.

Framework Shift

Before (mainstream XAI):            After (this paper):

 [ frozen dataset ]                  time -->
        |                            t1        t2        t3
        v                            |         |         |
 [ trained model ]                 [data_1]  [data_2]  [data_3]
        |                            |         |         |
        v                            v         v         v
 [ explainer ]                     [model_1] [model_2] [model_3]
        |                            |         |         |
        v                            v         v         v
 [ metric = 0.87 ]                 [expl_1]  [expl_2]  [expl_3]
        |                            :         :         :
        v                            +---------+---------+
   ship it                                     |
                                               v
 one number, one moment,               monitor the triple:
 no human, no clock              validity / feasibility /
                                 stability / human utility
                                 tracked as a trajectory

From a snapshot score to a monitored trajectory, the core shift is treating an explanation as a perishable claim about a moving model rather than a static artifact you compute once and grade.

Expert Assessment

Problem choice: Real gap, and unusually well-positioned. The XAI-evaluation complaint is well-trodden — there’s a decade of “we need better metrics” papers — but the intersection with data-stream mining and concept drift genuinely is thin, and Stefanowski is one of the few people credibly holding both threads (his stream-mining and imbalanced-data work sits on one side, the DetoxAI group’s XAI work on the other). The “co-evolution of data, models, and explanations” framing is the kind of thing that looks obvious in retrospect and nobody had cleanly named. If explanations end up in regulated pipelines (EU AI Act recourse obligations), explanation staleness becomes a compliance problem, not just an academic one.

Method maturity: This is a workshop invited-paper synthesis, not a methods contribution, and it should be read as an agenda rather than a solution. The one concrete technical thread — adapting counterfactuals to streams — is described as “experiences,” which usually means partial results. And there is a simpler baseline lurking that position papers like this tend to skate past: just *recompute the counterfactual on every model update. It’s expensive but conceptually trivial, and any paper proposing incremental explanation maintenance has to beat it on cost while matching validity. I’d want to see that comparison made explicit rather than assumed away.

Experimental integrity: Hard to stress-test from the abstract, and structurally the paper isn’t set up for it — three heterogeneous anchors (a system, a user study, stream experiments) each get a slice, which means none gets full treatment. The DetoxAI-as-motivation move is honest but circular by design, and the paper appears to acknowledge that rather than hide it, which I credit. The human-grounded study is the piece I’d scrutinize hardest: sample size, participant expertise, and whether the task was a genuine model-prediction task or a preference survey. “Users preferred explanation A” is a much weaker result than “users using explanation A predicted model failures better,” and the two get conflated in this literature constantly.

Writing quality: The predictable weakness of the invited-talk format: breadth over depth, and three case studies that don’t fully cohere into one argument. The section that would most elevate the paper is the counterfactual-under-drift material — that’s the original contribution, and it deserves a formal problem statement (define validity, feasibility, and stability as measurable quantities over a stream, then report them against a recompute-everything baseline) instead of an experience report. Give me that, and this becomes a paper people cite for a definition rather than for a sentiment.

Verdict: weak accept — the right diagnosis and a genuinely useful framing for a workshop audience, but as a position paper it identifies the gap far more sharply than it closes it.

Takeaways

Things worth stealing, concretely:

• The three-way decomposition of counterfactual rot. Validity (does the advice still flip the outcome under the current model), feasibility (is the recommended value still in-distribution), and stability (does similar input get similar advice across retrainings) are separable, measurable, and cheap to instrument. If you ship recourse or “what to change” advice in any production system with a retraining loop, you can log these three today. Most teams monitor model accuracy and nothing about explanation staleness.

• Explanation drift and concept drift are two clocks. The input distribution can move while the model is frozen (explanations shift, accuracy holds), or the model can be retrained with little change to what it attends to. Monitoring one does not give you the other. Useful framing for any monitoring stack.

• The circularity trap in verifying interventions. When you use an explanation method to prove that a debiasing or unlearning procedure worked, you’re validating an unvalidated tool with another unvalidated tool. The escape hatch is a behavioural check that doesn’t route through the explainer — held-out counterexamples, synthetic probes with known ground truth, targeted distribution shifts. Applies well beyond bias work: any time your evidence and your instrument share assumptions.

• Tier-check your evaluation before trusting a leaderboard. If a comparison of explanation methods reports only functionally-grounded metrics, treat the ranking as provisional. Rankings reverse at the human-grounded tier often enough that this is a real risk, not a caveat.

What’s not here: no ready-to-use algorithm, no benchmark, no code you can pull. This is a map of where to dig, not a shovel.