Necessary or Sufficient? Auditing LLM Explanations with Causal Interventions

Paper: 2609.05385 Authors: Urja Pawar, Rajitha Ramanayake, Nabeel Kemal, Ashwin Kandath, Owen O’Neill, Guillaume Bourgeon, Houssem Chatbri Categories: cs.AI

The Gap

When autonomous agents or decision-support pipelines are deployed in regulated enterprise domains — such as financial underwriting, wealth management, compliance auditing, or risk assessment — they are mandated to provide natural-language explanations. A model does not merely decide “Approve” or “Reject”; it provides a polite rationale citing the “Top 3 decisive factors” that drove its choice.

Compliance officers, human-in-the-loop supervisors, and prompt engineers rely heavily on these cited factors. If an agent states that a client was denied credit because of their “debt-to-income ratio,” operators assume that intervening on that specific factor would alter the decision.

Yet in modern deep learning, self-generated natural language is an ungrounded secondary token stream. A model’s self-explanation could be nothing more than a post-hoc rationalization — an articulate story woven after the decision was already finalized by invisible internal activations.

Prior evaluations of LLM explainability relied on human plausibility scoring (whether the explanation sounds sensible to a reader) or superficial feature attribution. What has been missing is strict, causal behavioral testing: measuring whether the model’s cited factors are mathematically necessary or sufficient to reproduce its decisions.

[ENTERPRISE DECISION PIPELINE] Client Profile -> LLM Agent -> Decision + Top-3 Explanations
                                      |
                                      v
     [COMMON ASSUMPTION] "The model's stated reasons reflect its actual internal drivers."
                                      |
    +---------------------------------+---------------------------------+
    v                                                                   v
[CAUSAL NECESSITY TEST]                                       [CAUSAL SUFFICIENCY TEST]
If factor X is truly decisive,                                If factor X is sufficient,
changing X *must* change the output.                          retaining X while scrambling other
                                                              features *must* preserve the output.
    |                                                                   |
    +---------------------------------+---------------------------------+
                                      v
[EMPIRICAL VERDICT] Spearman rank correlation is only ~0.35!
The factors an LLM boasts about have almost zero causal control over what it actually does.

The Increment

One sentence: Before this paper, enterprise agent pipelines trusted LLM-generated explanations for auditability and compliance; after it, BNY Mellon’s causal intervention suite demonstrates that an LLM’s self-reported factors correlate weakly (Spearman rho ~ 0.35) with actual causal drivers, proving that self-explanations cannot serve as compliance guarantees.

Core Mechanism

The authors formalize explanation faithfulness through causal black-box behavioral interventions applied to 8 frontier models spanning the Claude, GPT, and Gemini model families across two high-stakes tasks: financial advisor recommendations and prompt risk classification.

The framework measures two distinct philosophical and mathematical dimensions of causal explanation:

1. Necessity Auditing: For each factor cited by the model as a top driver, the authors systematically perturb only that attribute while holding all other inputs static. The Necessity Score measures the probability that modifying this factor flips the final decision: P(Output changes | Factor perturbed).
2. Sufficiency Auditing: The cited factor is frozen in place, while all other background information is masked, randomized, or perturbed. The Sufficiency Score measures how frequently the decision remains stable: P(Output preserved | Background scrambled).
3. Behavioral Rank Correlation: The model’s explicit self-ranked importance (Factor 1 > Factor 2 > Factor 3) is compared against the empirically measured Necessity and Sufficiency rankings via Spearman correlation.

   CAUSAL BEHAVIORAL INTERVENTION PIPELINE

                     [Original Decision Context]
                                  |
            +---------------------+---------------------+
            v                                           v
   [Perturb Cited Factor]                     [Scramble Background]
   (Freeze all other variables)               (Freeze only cited factor)
            |                                           |
            v                                           v
   [Did Decision Flip?]                       [Did Decision Hold?]
            |                                           |
            v                                           v
    NECESSITY SCORE: 0.349                     SUFFICIENCY SCORE: 0.354
            |                                           |
            +---------------------+---------------------+
                                  v
   [Conclusion: Weak Correlation with Stated Ranking!]
   Models cite factors that are neither necessary nor sufficient to cause their choices.

To illustrate this deception, consider a structural metaphor of a hiring manager in a job interview. The manager rejects a qualified candidate and explains in the official HR report: “The primary deciding factor was that the candidate lacks three years of Kubernetes experience.” The compliance department thinks the process is objective. But a forensic auditor tests the manager’s behavior: they send the exact same resume with Kubernetes added, and the manager still rejects the candidate. Then they send five mediocre resumes that also lack Kubernetes, and the manager hires two of them. The manager’s stated reason was a polite post-hoc excuse; their actual decision was driven by an unacknowledged cognitive bias.

Key Concepts

• Causal Faithfulness: The degree to which an AI system’s stated explanation corresponds to the actual causal mechanisms that determine its output behavior.
• Causal Necessity: A condition where an outcome would not have occurred in the absence of the cited factor.
• Post-Hoc Rationalization: An articulated justification generated after an inference step that sounds coherent to human readers but has zero mechanistic or counterfactual connection to the underlying model prediction.

Framework Shift

Before (Human Plausibility Paradigm):
Model Output + Explanation ---> [Human Reviewer] ---> "Sounds reasonable!" ---> Approved for Production
(Fatal flaw: humans mistake articulate prose for causal truth)

After (Causal Behavioral Intervention Paradigm):
Model Output + Explanation ---> [Automated Counterfactual Perturbations]
                                            |
                         +------------------+------------------+
                         v                                     v
                 [Necessity Check]                    [Sufficiency Check]
(Mathematically verifies whether mutating cited factors actually shifts the model's choices)

From accepting articulate natural-language justifications at face value to testing their counterfactual integrity through black-box interventions, the core shift is unmasking LLM self-explanations as ungrounded narrative generation.

Expert Assessment

Problem choice: Paramount for the financial and legal sectors. As enterprise deployments rush to comply with the EU AI Act and financial consumer protection laws, believing LLM self-explanations creates colossal liability.

Method maturity: Grounding the audit in the rigorous philosophical and mathematical definitions of counterfactual causality (Necessity and Sufficiency) elevates the work far above informal red-teaming.

Experimental integrity: Testing 8 frontier models across three leading LLM providers (Anthropic, OpenAI, Google) confirms that post-hoc rationalization is not an idiosyncrasy of one company’s RLHF tuning, but a structural property of autoregressive language generation.

Writing quality: Direct, sober, and unsparing. The distinction between human plausibility and algorithmic causality is drawn with brilliant clarity.

Verdict: strong accept — A vital reality check for enterprise AI governance and automated compliance architectures.

Takeaways

• Never use an LLM’s self-generated explanation as a legal, financial, or safety audit trail.
• An explanation that sounds 100% convincing to a human auditor often has a Spearman correlation below 0.35 with what actually moved the model’s weights.
• True explainability requires active counterfactual intervention suites (like SHAP or behavioral perturbation frameworks), not asking the model “Why did you do that?”