Gradient Probes: Finding Hidden Biases in Vision Models Without Labels

Paper: 2605.28780 Authors: Thomas Vitry, Kieran Edgeworth, Stefan Wermter, Jae Hee Lee Categories: cs.CV, cs.LG

The Gap

Existing bias mitigation methods hit a wall when models are already deployed. Methods like reweighting, group DRO, or adversarial debiasing require knowing which attributes are spurious (gender, background, color) and having labels for them during training. But in production, you often discover bias after deployment, when retraining is expensive or infeasible, and you don’t know which concepts the model is exploiting. Prior post-hoc analysis tools like concept activation vectors (CAVs) still need human-annotated concept examples. The field lacks a way to audit frozen models for unknown biases using only standard class labels.

Problem: Deployed model fails on subgroups
    |
    v
Existing solutions require:
  - Spurious attribute labels (gender, background)
  - Retraining from scratch
  - Known bias concepts upfront
    |
    v
Gap: No post-hoc method for unknown biases in frozen models
    |
    v
This paper's assumption:
  Bias concepts leave gradient signatures on misclassified examples
    |
    v
Method: Decompose activations -> rank concepts by gradient interaction
    |
    v
Evidence: Recovers known spurious cues (color, background)
         Improves worst-group accuracy without retraining
    |
    v
Conclusion: Gradient probes identify actionable bias in frozen models

The Increment

One sentence: Before this paper, auditing deployed models for unknown biases required retraining or bias labels; after, you can identify and suppress spurious concepts in frozen models using only class labels.

Core Mechanism

The method operates in three stages. First, for each target class, collect image patches that the frozen model predicts as that class. Extract intermediate layer activations from these patches and apply non-negative matrix factorization (NMF) to decompose them into a bank of interpretable concept vectors—think of these as the model’s internal vocabulary for that class.

Second, identify which concepts are spurious by examining misclassified examples. For false negatives (should be class A but predicted otherwise), backpropagate gradients and measure how much each concept vector would need to be activated to correct the mistake. For false positives (predicted as class A but shouldn’t be), measure how much each concept needs to be suppressed. Bias concepts show a characteristic signature: they get strongly activated when fixing false negatives and strongly suppressed when fixing false positives.

Third, rank concepts by this gradient-based bias score. At inference time, suppress the top-ranked concepts by zeroing out their contributions in the activation space. No parameter updates, no retraining—just intervention on intermediate representations.

Input image -> Frozen model -> Intermediate activations
                                        |
                                        v
                    NMF decomposition into concept vectors
                    (one bank per class)
                                        |
                                        v
                    Misclassified examples -> Backprop gradients
                                        |
                                        v
                    Bias score = activation on FN - activation on FP
                                        |
                                        v
                    Rank concepts by bias score
                                        |
                                        v
                    Inference: Suppress top-ranked concepts

Think of the model’s intermediate layer as a mixing board with many sliders, each controlling a concept (texture, shape, color, background). NMF discovers what these sliders represent by analyzing which combinations fire for each class. Now imagine you have recordings of mistakes—songs that came out wrong. You play them backward (backpropagation) to see which sliders were set incorrectly. Bias concepts are the sliders that consistently need to be turned up when the model missed something (false negative) and turned down when it hallucinated something (false positive). Once you’ve identified the problematic sliders, you can lock them at zero during playback without rewiring the entire mixing board.

Key Concepts

• Non-negative Matrix Factorization (NMF): Imagine you have a spreadsheet where rows are image patches and columns are neuron activations. NMF finds a small set of “basis patterns” (concept vectors) such that each patch’s activations can be written as a non-negative weighted sum of these patterns. Unlike PCA, which allows negative weights and produces abstract axes, NMF’s non-negativity constraint yields additive, part-based representations—like decomposing a face into “eyes,” “nose,” “mouth” rather than “face minus anti-face.” In this paper, NMF decomposes activations from patches of a given class into interpretable concept directions that the model uses to recognize that class.

• Gradient-based bias estimation: When the model misclassifies an example, backpropagated gradients tell you how to adjust activations to fix the mistake. For a false negative (missed a bird), gradients point toward increasing activations that would make the model say “bird.” For a false positive (hallucinated a bird), gradients point toward decreasing those activations. A spurious concept (like “water background”) will show high positive gradients on false negatives (waterbirds the model missed) and high negative gradients on false positives (landbirds the model wrongly called waterbirds). The bias score captures this asymmetry: concepts that need to be amplified for one type of error and suppressed for the other are likely spurious shortcuts.

• Post-hoc intervention on frozen models: Most debiasing methods retrain the model with modified objectives or data. This paper intervenes at inference time by identifying spurious concept directions in activation space and projecting them out—like noise cancellation for bias. The model’s parameters stay frozen; you’re just filtering its internal representations. This is crucial for deployed models where retraining is prohibitively expensive or when you discover new biases after deployment. The tradeoff: you can’t fundamentally change what the model learned, only suppress specific decision pathways.

Framework Shift

Before (mainstream approach):        After (this paper):

Training time:                       Training time:
  Data + bias labels                   Data + class labels only
       |                                    |
       v                                    v
  Reweight / adversarial               Train normally
       |                                    |
       v                                    v
  Debiased model                       Standard model
                                            |
                                            v
                                       Deployment
                                            |
                                            v
                                       Audit phase:
                                         Collect patches
                                         NMF -> concepts
                                         Gradient probes
                                            |
                                            v
                                       Suppress at inference

Requires: Known biases upfront       Requires: Only class labels
Flexibility: None after training     Flexibility: Iterative auditing

[One sentence: From baking bias mitigation into training to applying it as a post-deployment filter, the core shift is from preventive to detective.]

Expert Assessment

Problem choice: Real gap. The scenario—deployed model, unknown bias, no retraining budget—is common in industry but underserved in research. Most bias work assumes you know what to look for and can retrain. This paper targets the harder case.

Method maturity: The gradient-based bias score is clever but not deeply justified. Why should the asymmetry between false negatives and false positives specifically isolate spurious concepts rather than just hard-to-learn features? The paper shows it works empirically but doesn’t provide theoretical grounding. NMF for concept discovery is standard; the novelty is in the ranking criterion. The method is simple enough to be practical, which is a strength.

Experimental integrity: Baselines are fair—they compare against other post-hoc methods (CAVs, TCAV) and show their approach works without concept labels. The datasets (Colored MNIST, Waterbirds, CelebA) are standard bias benchmarks. However, the improvements are measured on worst-group accuracy, which can be gamed by sacrificing overall accuracy. The paper reports both, which is honest, but the tradeoff curves could be more prominent. One concern: the method assumes misclassified examples are available and representative of bias, which may not hold if the audit dataset is small or unbalanced.

Writing quality: The paper is clear in the method section but rushes through limitations. Section 5 (experiments) could better separate “does it find the right concepts?” from “does suppression help?” The related work section is thorough but could be trimmed. The biggest missed opportunity: no discussion of when the method fails. What if bias concepts are entangled with valid features? What if gradients are noisy? A failure mode analysis would elevate this from a technique paper to a methods paper.

Verdict: weak accept — Addresses a real problem with a practical solution, but the theoretical justification is thin and the experimental scope is narrow (three datasets, all with known ground-truth biases). Needs broader evaluation on cases where the spurious concept is truly unknown.

Takeaways

Gradient asymmetry as a bias signal: The idea that spurious concepts show opposite gradient patterns on false negatives vs. false positives is transferable. You could apply this to debug other model failures—not just bias, but any systematic error where the model overuses a feature in some contexts and underuses it in others.

NMF for interpretable decomposition: If you need human-interpretable directions in activation space, NMF’s non-negativity constraint is underused compared to PCA. It’s worth trying when you want part-based, additive concepts rather than abstract axes.

Inference-time intervention: The broader lesson is that you don’t always need to retrain. If you can identify problematic directions in representation space, you can filter them out at inference. This applies beyond bias—think adversarial robustness, domain adaptation, or removing sensitive attributes for fairness.

Audit-then-act workflow: The paper demonstrates a two-phase approach: first audit the model to discover issues, then intervene. This is more realistic than assuming you know all problems upfront. The workflow itself—collect examples, decompose, rank, suppress—is a template for post-hoc model debugging.