CAM's Family Tree: A Method-Centered Map of Visual Explanation from CNNs to Foundation Models

Paper: 2608.12299 Authors: AmirHossein Eshghi, Hamid Saadatfar, Seyyed Ali Hoseini, AmirMohsen Eshghi, Siavash Arjomand Bigdel Categories: cs.AI, cs.CV

The Gap

Here’s the situation this paper walks into. CAM (2016) needed a global-average-pooling head, so it only worked on a narrow class of CNNs. Grad-CAM removed that constraint by using gradients as channel weights. Grad-CAM++ and XGrad-CAM patched gradient saturation and averaging artifacts. Score-CAM and Ablation-CAM threw gradients out entirely and scored channels by measuring what happens to the output when you mask or ablate them. LayerCAM and HiResCAM attacked the resolution problem — last-layer feature maps are 7x7 and upsampling them to 224x224 is a lie dressed as a heatmap. Then ViTs arrived and the whole “last convolutional layer” premise stopped existing, so token-attribution methods appeared. Then CLIP, DINO, and SAM arrived and the premise of “a target class score” also weakened, because now you might be explaining a text-image similarity or a self-supervised feature cluster.

So the literature is not a list, it’s a chain of patches. But it’s read as a list. Existing surveys tend to either group everything under “XAI methods” at a level too abstract to be useful, or benchmark a handful of CAM variants on one dataset with one faithfulness metric. Neither tells you why Score-CAM exists or what it failed to fix. The specific gap this paper targets: nobody has organized the CAM family by the structural properties that actually determine when a method is applicable — what signal it extracts, what architecture it assumes, and what it optimizes for — and nobody has traced the gap-to-gap dependency between methods.

Problem: 10 years of CAM variants read as a flat list.
         "Which one should I use?" has no principled answer.
              |
              v
Assumption: every method can be located on 3 readable axes
            (attribution mechanism / architectural dependence
             / evaluation objective), and each one exists
            because it names a flaw in an earlier one.
              |
              v
Method: strict corpus of 57 method-centered papers, 2016+
        - build the 3-axis taxonomy
        - review gradient CAMs, hybrid CAMs, model-aware CAMs
        - for each: contribution AND the gap it leaves open
              |
              v
Evidence: the corpus-wide trend is one-directional
          one class score --> comparative / contrastive
          one conv layer  --> multi-layer aggregation
          deterministic   --> probabilistic, causal
          CNN-only        --> tokens, CLIP, DINO, SAM
          ... but faithfulness, localization, robustness,
              cost, and human trust each use different
              protocols across papers.
              |
              v
Conclusion: mechanism innovation is healthy;
            evaluation is the actual bottleneck.
            You cannot rank these methods because
            the field never agreed on the ruler.

The Increment

One sentence: Before this paper, the CAM literature was a decade-long pile of variants you had to read chronologically to understand; after it, you have a three-axis coordinate system that tells you which methods are even comparable to each other, plus an explicit map of which gap each method opened and which later method closed it.

Core Mechanism

The “method” here is a classification scheme, so the internals are the axes. Axis one, attribution mechanism: what signal does the method turn into a weight? Gradient-based methods use backprop signal at some layer. Gradient-free score methods use forward passes on perturbed inputs. Ablation methods delete a channel and measure the output drop. Causal methods try to isolate the effect of a region while holding confounders fixed. Distribution-comparison methods (the foundation-model-era group) compare a sample’s features against a reference distribution rather than reading a class logit at all. This axis predicts computational cost almost perfectly — one backward pass versus hundreds of forward passes.

Axis two, architectural dependence: does the method require GAP (original CAM), any convolutional stack (Grad-CAM family), attention plus residual token streams (transformer attribution), or an external foundation model as a prior (CLIP text alignment, DINO features, SAM masks)? This axis is the applicability filter. A practitioner with a ViT-B/16 can immediately discard two-thirds of the corpus.

Axis three, evaluation objective: is the method trying to be faithful (deletion/insertion curves, sanity checks), to localize (pointing game, IoU against boxes, WSOL/WSSS scores), to be robust (stability under noise or adversarial shift), to be cheap, or to be trusted by humans? This is where the paper’s negative finding lives. Methods that win on localization frequently lose on faithfulness, because a sharp, object-shaped mask looks like a good explanation to a human and to an IoU metric whether or not it reflects the model’s actual computation. The review’s per-method template — contribution, remaining gap, successor that addresses it — turns the corpus into a directed graph instead of a bibliography.

        [ corpus: 57 method papers, 2016 .. present ]
                            |
        +-------------------+-------------------+
        |                   |                   |
        v                   v                   v
 AXIS 1: MECHANISM   AXIS 2: ARCH DEP.   AXIS 3: OBJECTIVE
 . gradient          . GAP-CNN only      . faithfulness
 . grad-free score   . any CNN           . localization
 . ablation          . ViT tokens        . robustness
 . causal / debias   . CLIP / DINO / SAM . cost
 . distributional    . model-agnostic    . human trust
        |                   |                   |
        +-------------------+-------------------+
                            |
                            v
              per-method record template:
        [ what it adds ] -- [ what it still misses ]
                            |
                            v
                     GAP CHAIN (excerpt)
   CAM
    | needs GAP head, must retrain
    v
   Grad-CAM
    | gradient saturation, coarse 7x7 maps
    v
   Grad-CAM++ / XGrad-CAM ....... fix the weighting
    | gradients unreliable / shattered
    v
   Score-CAM / Ablation-CAM ..... drop gradients, pay in FLOPs
    | still last-layer, still low resolution
    v
   LayerCAM / HiRes-CAM ......... multi-layer, sharper
    | assumes convolutions exist
    v
   token attribution (ViT) ...... attention + residual flow
    | assumes a labeled class score exists
    v
   CLIP / DINO / SAM-based ...... open-vocabulary, label-free
    | no shared benchmark to prove any of this
    v
   [ OPEN: unified evaluation protocol ]

Think of it as a relay race where nobody agreed where the finish line is. Each method is a runner. The baton is a specific named defect — “you need to retrain the network,” “your gradients saturate,” “your map is 7x7,” “your model has no convolutions.” A runner accepts the baton, fixes that one thing, and in the act of running creates a new defect, which becomes the baton for the next leg. That’s the gap chain, and it’s why the literature is genuinely sequential rather than parallel.

The three axes are the lanes. Runners in the gradient lane can be timed against each other; a runner in the CLIP-distribution lane is on a different track entirely, and comparing their times is meaningless — which is exactly the mistake papers make when they benchmark Score-CAM against a CLIP-based explainer on one deletion curve. And the finish line is the evaluation objective: some runners are sprinting toward faithfulness, some toward IoU with a bounding box, some toward “a radiologist said it looked right.” They all get medals, because each race declared its own tape. The review’s contribution is not a new runner. It’s drawing the lanes and pointing out that there are five finish lines, so the leaderboard everyone cites is fiction.

Key Concepts

• Why gradients work as “importance” at all: Take the last convolutional layer. It outputs, say, 512 feature maps, each a 7x7 grid — think of them as 512 detector sheets, one lighting up for stripes, one for round shapes, one for grass texture. To explain “zebra,” you need to know which sheets mattered. The gradient of the zebra score with respect to a sheet answers “if this sheet got slightly brighter everywhere, how much would the zebra score rise?” Average that over the sheet, use it as the sheet’s weight, sum all 512 weighted sheets, and you get a 7x7 heatmap. That’s Grad-CAM in one paragraph. The catch: gradients answer an *infinitesimal question. If the zebra score is already saturated at 0.999, nudging the stripe detector changes nothing, so the gradient is near zero and the stripe sheet gets weight zero — even though stripes are the entire reason for the prediction. This one failure mode generated a whole subfamily of methods.

• Gradient-free scoring, and what it costs: Instead of asking the infinitesimal question, ask the blunt one. Take feature sheet number 137, turn it into a mask over the image, show the model only that region, and see what the zebra score becomes. High score means sheet 137 was carrying real evidence. Do this for all 512 sheets, use the scores as weights. No gradients, no saturation, no shattered-gradient noise. The price is 512 forward passes instead of one backward pass — a factor of hundreds in latency. This is the paper’s cleanest illustration of the mechanism axis being a cost axis in disguise: the same explanation quality can be bought with math or bought with compute.

• Faithfulness vs. localization is a real conflict, not a metric detail: Faithfulness asks “does this heatmap reflect what the model actually used?” You test it by deleting the highlighted pixels and checking that the prediction collapses. Localization asks “does this heatmap sit on the object?” You test it against a human-drawn box. These diverge whenever the model is right for the wrong reason. If a classifier detects “boat” from water texture, the faithful explanation highlights water and scores terribly on IoU with the boat box; a method that produces crisp boat-shaped masks scores beautifully on localization while actively hiding the model’s real behavior. Pretty explanations and honest explanations are in tension, and the review’s finding that these two objectives are measured in separate papers with separate protocols means the field has been able to avoid confronting this.

Framework Shift

Before (mainstream approach):          After (this paper):

  one image                              one image
     |                                      |
     v                                +-----+-----+-----+
  one CNN                             |     |     |     |
     |                                v     v     v     v
  last conv layer only              layer1 layer2 ... tokens
     |                                |     |     |     |
     v                                +-----+--+--+-----+
  one class logit                              |
     |                                         v
     v                            compare: class A vs class B,
  weights = mean gradient          sample vs reference distribution,
     |                             image vs text prompt (CLIP),
     v                             region vs SAM mask prior
  7x7 map --> upsample 224x224                 |
     |                                         v
     v                              probabilistic / causal weight
  "looks about right"                          |
                                               v
                                    evaluate on: faithfulness?
                                    localization? robustness?
                                    cost? human trust?
                                               |
                                               v
                                      ( no shared protocol )
                                        <-- the open gap

One sentence: from explaining one class score in one low-resolution CNN layer to comparative, multi-layer, probabilistic, token- and foundation-model-aware attribution, the core shift is that an explanation stopped being a single readout and became a comparison — and the field’s bottleneck moved from producing heatmaps to agreeing on how to judge them.

Expert Assessment

Problem choice: The gap is real but modest. CAM surveys exist, and general XAI surveys exist by the dozen. What’s genuinely underserved is the seam this paper sits on: the transition from “explain a CNN classifier’s logit” to “explain a CLIP alignment or a DINO feature,” where the original CAM formulation’s assumptions — a class score, a convolutional stack, a GAP layer — have all quietly dissolved. Someone needed to write that down. The three-axis taxonomy is a sensible organizing move and the gap-chain framing is the best thing in the paper, because it captures something a flat taxonomy can’t: this literature is causally sequential. The observation that evaluation is fragmented is correct, well-known, and has been stated in nearly every XAI survey since 2020, so credit for the diagnosis is thin.

Method maturity: A taxonomy is not a method, and this one is defensible rather than surprising. Mechanism / architecture / objective is close to what a thoughtful practitioner would reach for unaided. The 57-paper “strict corpus” is the weak joint. Strict is good for reproducibility, but 57 is small for a decade-plus of one of the most-copied ideas in computer vision, and the abstract doesn’t reveal the inclusion rule. If the filter is “papers proposing a named CAM variant,” it’s defensible. If it’s a keyword search with a citation floor, then the corpus is a convenience sample and the “clear trend” claim is partly an artifact of what got included. Reviews live and die on this and it should be adjudicated in section 2, not asserted.

Experimental integrity: There are no experiments, which is legitimate for a review but sets a ceiling. The paper’s central finding — evaluation is fragmented and protocols aren’t comparable — is exactly the kind of claim that would be devastating if demonstrated and is merely persuasive when asserted. A single table reimplementing eight representative methods under one deletion protocol, one pointing game, and one cost measurement would have shown rank inversions across metrics and converted the argument from “these papers use different rulers” to “here is the specific method whose reported superiority evaporates.” That table is the missing 30% of this paper. Related risk: the taxonomy is unvalidated. If two independent readers can’t place the same method on the same axis coordinates, it’s a narrative device rather than a classification.

Writing quality: The abstract is well-organized and the per-method “contribution plus remaining gap” template is a discipline most surveys lack — it forces the authors to say something evaluative rather than paraphrase. Where I’d expect corners cut: the foundation-model section. CLIP/DINO/SAM-based explanation is the newest and thinnest part of the literature, it’s where the paper claims its novelty, and it’s also where a review is most likely to devolve into three-sentence summaries of ten papers without a shared frame. If one section rewrite would elevate the whole thing, it’s that one — specifically, making explicit what “explanation” even means when there is no class, and how faithfulness is defined when the thing being explained is a similarity in an embedding space. That question is unanswered in the field and the review is well-positioned to sharpen it.

Verdict: weak accept — a useful, well-structured map of a fragmented literature with a genuinely good gap-chain framing, held back by a small unjustified corpus, an unvalidated taxonomy, and a headline finding about evaluation that is asserted rather than demonstrated.

Takeaways

Concrete things worth stealing:

• The gap-chain template as a literature-review tool. “Contribution, remaining gap, successor that addresses it” is a better structure than “contribution” alone, and it generalizes to any field where methods are patches on prior methods — optimizers, PEFT variants, RAG architectures, quantization schemes. It converts a bibliography into a directed graph, and the leaf nodes are your open problems.
• Separate the applicability axis from the quality axis. Before comparing two explanation methods, check whether they even assume the same architecture and the same explanation target. Half the CAM comparisons in the wild are category errors. Same trap exists in agent benchmarks and in retrieval evaluation.
• Mechanism is cost in disguise. Gradient-based versus perturbation-based is not primarily an accuracy tradeoff, it’s one backward pass versus O(channels) forward passes. If you’re picking an explainer for a production inference path, that axis decides for you before quality does.
• Distrust localization scores as explanation quality. A crisp object-shaped mask is evidence about the metric, not about the model. If your model is right for the wrong reason, the honest explanation will score badly on IoU. Always pair a localization metric with a deletion/insertion metric, and treat disagreement between them as information rather than noise.
• The reference-distribution reframe. The foundation-model-era move — explain by comparing a sample’s features against a reference distribution instead of reading a class logit — transfers well beyond vision. It’s how you get explanations for models with no label space at all: embedding models, anomaly detectors, self-supervised encoders.

What not to expect: no new method, no benchmark, no numbers you can cite. Read it to orient yourself in the CAM family and to steal the taxonomy; don’t read it to decide which explainer is best, because the paper’s own conclusion is that the literature can’t answer that yet.