Unlocking Patch-Level Features for CLIP-Based Class-Incremental Learning

Paper: 2605.13835 Authors: Hao Sun, Zi-Jun Ding, Da-Wei Zhou Categories: cs.CV

The Gap

CLIP-based class-incremental learning (CIL) has become the dominant paradigm because CLIP’s pre-trained representations generalize well to new classes. But current methods only align global image embeddings (the [CLS] token) with global text prompts (the [EOS] token). They throw away the patch tokens — the 196+ local features that CLIP’s vision encoder produces for every image. When recognizing a rabbit, those patches encode “long ears” and “fluffy tail,” but existing methods never look at them. The gap: we’re using 1/200th of CLIP’s visual information.

Problem: CLIP-CIL uses only global tokens
   |
   v
Observation: Patch tokens encode local semantics (ears, tail, texture)
   |
   v
Assumption: Aligning patches with semantic descriptions improves recognition
   |
   v
Method: SPA = GPT-generated descriptions + optimal transport alignment
   |
   v
Evidence: SOTA on CIFAR-100, ImageNet-R, ImageNet-A (Table 1)
   |
   v
Conclusion: Patch-level alignment > global-only alignment for CIL

The Increment

One sentence: Before SPA, CLIP-based CIL used only global image-text alignment; after SPA, we align discriminative local patches with semantic descriptions, unlocking CLIP’s full representational capacity.

Core Mechanism

SPA has three stages. First, for each class (say, “rabbit”), collect representative images and feed them to GPT-5 with visual guidance to generate semantic descriptions like “long ears, fluffy tail, small nose.” Second, use these descriptions to select discriminative patch tokens from CLIP’s vision encoder — patches that correspond to “ears” or “tail” rather than background. Third, align selected patch tokens with semantic tokens from the descriptions using optimal transport, creating a structured cross-modal mapping.

To prevent forgetting old classes, SPA stores Gaussian statistics (mean, covariance) for each learned class and samples pseudo-features during training to calibrate old-class representations. Task-specific projectors adapt the frozen CLIP backbone to each incremental task without interfering with prior knowledge.

Input Image --> CLIP Vision Encoder --> [CLS] + 196 patch tokens
                                              |
                                              v
                                    Select discriminative patches
                                    (guided by GPT descriptions)
                                              |
                                              v
                                    Optimal Transport Alignment
                                    patches <--> semantic tokens
                                              |
                                              v
                                    Task-specific projector --> Classification
                                              ^
                                              |
                                    Pseudo-features from Gaussian
                                    (old classes, anti-forgetting)

Think of SPA as a museum curator organizing an exhibit. The global [CLS] token is like the museum’s one-sentence mission statement — useful but generic. The patch tokens are individual artifacts in storage. GPT-5 acts as a domain expert who writes detailed catalog entries (“long ears, fluffy tail”). The curator (SPA) uses these entries to select which artifacts to display and arranges them in correspondence with the catalog descriptions (optimal transport). When new exhibits arrive (new classes), the curator adds a new wing (task-specific projector) without rearranging the old exhibits. To remember old exhibits no longer on display, the curator keeps statistical records (Gaussian distributions) and occasionally recreates them (pseudo-features) to ensure staff don’t forget what a rabbit looks like.

Key Concepts

• Optimal Transport Alignment: Imagine you have two sets of objects — patch tokens from an image and semantic tokens from a text description. Optimal transport finds the “cheapest” way to move mass from one set to the other, where cost is measured by feature similarity. In SPA, this creates a structured mapping: the “long ears” patch gets aligned with the “long ears” semantic token, not randomly. It’s like matching puzzle pieces by shape rather than forcing them together. The result is a cross-modal correspondence that respects semantic structure, unlike naive dot-product attention which can align anything to anything.

• Gaussian Pseudo-Features: When learning new classes, the model risks forgetting old ones (catastrophic forgetting). SPA stores the mean and covariance of each old class’s feature distribution. During training on new classes, it samples synthetic features from these Gaussians and treats them as real old-class examples. This is like a teacher showing students old exam questions while teaching new material — it keeps old knowledge active. The Gaussian assumption is strong but practical: it compresses an entire class into two statistics, making storage feasible for hundreds of classes.

• Task-Specific Projectors: CLIP’s backbone is frozen to preserve pre-trained knowledge. But each incremental task (say, learning animals, then vehicles) has different feature requirements. Task-specific projectors are small learnable layers added per task that adapt CLIP’s features without modifying the backbone. Think of them as interchangeable camera lenses: the camera body (CLIP) stays the same, but you swap lenses (projectors) depending on whether you’re shooting wildlife or architecture. This prevents new tasks from overwriting old knowledge while still allowing adaptation.

Framework Shift

Before (mainstream approach):        After (this paper):

Image --> CLIP --> [CLS]             Image --> CLIP --> [CLS] + patches
                     |                                      |
                     v                                      v
Text  --> CLIP --> [EOS]             Text  --> GPT  --> descriptions
                     |                                      |
                     v                                      v
              Align globally                    Select + align patches
                     |                           (optimal transport)
                     v                                      |
              Classification                                v
                                                Task projector + Gaussian
                                                      |
                                                      v
                                                Classification

From global token alignment to structured patch-level alignment, the core shift is exploiting CLIP’s full spatial resolution instead of collapsing it into a single vector.

Expert Assessment

Problem choice: Real gap. CLIP’s patch tokens are information-rich but genuinely underutilized in CIL. The observation that local features (ears, tail) provide complementary evidence is obvious in hindsight but wasn’t systematically exploited. This sits at the intersection of two hot areas (CLIP adaptation and CIL), which is strategic but not manufactured.

Method maturity: Mixed. The GPT-5 description generation is clever but feels like overkill — do you really need a frontier LLM to generate “long ears, fluffy tail”? A simpler attribute lexicon might suffice. Optimal transport is theoretically elegant but computationally expensive; the paper doesn’t discuss runtime. The Gaussian pseudo-features are standard practice (see iCaRL, 2017), not novel. The core insight (use patches) is strong; the execution adds complexity that may not all be necessary.

Experimental integrity: Solid baselines (PROOF, SimpleCIL, EASE) and standard benchmarks (CIFAR-100, ImageNet-R/A). Table 1 shows consistent gains (2-4% average accuracy improvement). Ablations (Table 2) validate each component. However, no analysis of failure cases or computational cost. The claim of “state-of-the-art” is supported but the margins are incremental, not transformative.

Writing quality: The abstract and introduction are clear. Section 3 (method) is dense and would benefit from a running example. The optimal transport formulation (Eq. 3-4) is presented without intuition — a toy example with 3 patches and 3 semantic tokens would help. The related work section is thorough but reads like a literature dump. Rewriting Section 3.2 with a concrete walkthrough would elevate the paper significantly.

Verdict: weak accept — The core idea (patch-level alignment) is sound and underexplored, and the results are convincing. But the method feels over-engineered (GPT-5 for descriptions?) and the writing doesn’t make the contribution as accessible as it could be. It’s a solid incremental advance, not a paradigm shift.

Takeaways

Steal the patch selection strategy: Even outside CIL, if you’re using CLIP, don’t discard patch tokens. Use semantic guidance (descriptions, attributes, or even class names) to select discriminative patches. This is applicable to few-shot learning, domain adaptation, or any task where you want fine-grained alignment.

Optimal transport for structured alignment: When aligning two sets of features (visual-textual, source-target, teacher-student), optimal transport enforces semantic correspondence better than naive attention. The cost is higher compute, but the gain is interpretability and structure. Useful for cross-modal retrieval or knowledge distillation.

Gaussian statistics for replay: Storing class-wise Gaussians is a lightweight alternative to exemplar replay in continual learning. If memory is tight, this compresses an entire class into two matrices. The assumption (Gaussian distribution) is strong but works surprisingly well in practice.

Don’t over-engineer: The GPT-5 description generation is the weakest link. If you’re adapting this, try simpler alternatives first (WordNet attributes, template-based descriptions). The core insight (patches + alignment) doesn’t depend on a frontier LLM.