Why MLLMs Look Bad at Classification: It's the Benchmark, Not the Model

Paper: 2603.06578 Authors: Nikita Kisel, Illia Volkov, Klara Janouskova, Jiri Matas Categories: cs.CV

The Gap

The field has been comparing MLLMs (like GPT-4V, Gemini) against supervised classifiers (ResNets, ViTs) and getting wildly inconsistent results. Some studies show MLLMs crushing it, others show them embarrassingly behind. The problem? Nobody’s been honest about how broken the evaluation setup is.

Prior work uses three main protocols: multiple-choice (pick from N options), open-world (generate any class name), and zero-shot (classify without examples). Each protocol has hidden landmines. Multiple-choice inflates scores when distractors are too easy. Open-world tanks scores because “German Shepherd” doesn’t match “German shepherd dog” in the ground truth. And everyone ignores that batch size, image order, and text encoder choice swing accuracy by several percentage points.

Worse, ImageNet labels are noisy. A “tiger cat” image might actually show three cats, but the label says singular. MLLMs see all three and get marked wrong for being more accurate than the ground truth.

Conflicting benchmarks
        |
        v
Hypothesis: protocols are broken, not models
        |
        +---> Fix 1: Handle out-of-vocabulary outputs
        +---> Fix 2: Use harder distractors  
        +---> Fix 3: Improve output mapping
        +---> Fix 4: Control overlooked variables
        |
        v
Test on clean labels (ReGT reannotation)
        |
        v
Result: MLLM gap shrinks by ~11%, protocol matters more than architecture

The Increment

One sentence: Before, we thought MLLMs were bad classifiers; now we know the benchmarks were lying about their performance.

Core Mechanism

This isn’t a new model architecture—it’s a forensic audit of how we measure models. The authors systematically fix evaluation protocols across three settings.

For multiple-choice, they replace random distractors with semantically similar ones (don’t compare “dog” against “airplane”—compare it against “wolf” and “coyote”). For open-world, they build a fuzzy matcher that maps “German Shepherd” to “German shepherd dog” instead of discarding the answer. For zero-shot, they control variables everyone ignores: batch size (affects context window), image ordering (affects attention patterns), and text encoder choice (CLIP vs sentence transformers).

Then they reannotate 625 ImageNet classes with multiple correct labels per image. An image tagged “tiger cat” might actually contain “tiger cat,” “tabby cat,” and “Egyptian cat”—all valid. This is ReGT (Re-annotated Ground Truth).

Input: Image + Evaluation Protocol
   |
   v
+----------------------------------+
| Protocol Fixes:                  |
|  - Semantic distractors (MC)     |
|  - Fuzzy output mapping (OW)     |
|  - Controlled variables (ZS)     |
+----------------------------------+
   |
   v
+----------------------------------+
| ReGT: Multi-label ground truth   |
|  - Human reannotation            |
|  - MLLM-assisted validation      |
+----------------------------------+
   |
   v
Output: Corrected accuracy scores

Think of this like debugging a thermometer. You’ve been measuring patients’ temperatures and concluding everyone has a fever. But the thermometer was miscalibrated—it reads 2 degrees high. This paper recalibrates the thermometer (fixes protocols) and also checks if the patients actually have fevers (cleans labels). Turns out most patients are fine; your measurement tool was broken.

The protocol fixes are the recalibration. The ReGT dataset is checking actual patient health. The “patients” (MLLMs) weren’t sick—your diagnostic tool was faulty.

Key Concepts

• Out-of-vocabulary (OOV) outputs: When an MLLM generates “German Shepherd” but the class list only contains “German shepherd dog,” old protocols discard this answer entirely. It’s like a teacher marking your essay wrong because you wrote “automobile” instead of “car”—technically different strings, semantically identical. The fix: fuzzy matching that maps synonyms and handles capitalization, plurals, and minor variations. This alone recovers several percentage points.

• Distractor strength in multiple-choice: If you ask “Is this a dog?” and the choices are (A) dog, (B) airplane, (C) refrigerator, even a random guesser scores 33%. But if choices are (A) dog, (B) wolf, (C) coyote, you’re actually testing visual discrimination. Old benchmarks used weak distractors, making MLLMs look better than they are. Strong distractors reveal true capability. It’s the difference between a kindergarten quiz and a college exam.

• Multi-label ground truth: ImageNet assigns one label per image, but reality is messier. A photo of a park might contain a bench, a tree, a dog, and a person. If the label says “park bench” and an MLLM says “golden retriever,” it’s not wrong—it’s seeing something the label ignored. ReGT captures this by having annotators mark all valid objects. This shifts evaluation from “did you guess the one arbitrary label?” to “did you see something actually present?”

Framework Shift

Before (mainstream approach):        After (this paper):

Model --> Prediction                 Model --> Prediction
           |                                    |
           v                                    v
    String match with                    Fuzzy semantic match
    single ground truth                       |
           |                                   v
           v                              Multi-label GT
    Accuracy score                             |
                                               v
                                         Accuracy score
                                         
Assumption: labels are perfect         Assumption: labels are noisy,
            protocols are neutral                protocols have bias

From “trust the benchmark” to “audit the benchmark,” the core shift is treating evaluation as a system with its own failure modes, not a neutral oracle.

Expert Assessment

Problem choice: This is a real gap, and an important one. The field has been arguing about MLLM capabilities based on broken measurements. It’s not the sexiest problem—no new architecture, no SOTA claims—but it’s the kind of unglamorous work that prevents entire subfields from chasing ghosts. Sits squarely in the “someone had to do this” category.

Method maturity: Mostly careful engineering, not deep insight. The fixes are obvious once you see them (fuzzy matching, better distractors, controlled variables), but nobody bothered to do the work systematically. The ReGT reannotation is the real contribution—625 classes is substantial effort. No overlooked simpler approaches; this is just rolling up sleeves and fixing things properly.

Experimental integrity: Baselines are fair, numbers hold up. The +10.8% gain from clean labels is striking and well-documented. One minor flag: the paper doesn’t deeply explore why certain MLLMs benefit more from label corrections than others, though they note models less reliant on supervised signals are most sensitive. Would’ve liked more analysis there.

Writing quality: The paper front-loads results and buries methodology details, which makes it punchy but harder to reproduce. The related work section is thin—doesn’t engage deeply with prior evaluation critiques in other domains (NLP has been having this conversation for years). Rewriting the “protocol fixes” section with clearer before/after examples would help practitioners adopt these methods.

Verdict: weak accept — Solid empirical work that corrects widespread measurement errors, but lacks theoretical depth or surprising insights.

Takeaways

If you’re evaluating any generative model on classification:

1. Build a fuzzy matcher for outputs. Don’t discard answers because of capitalization or synonyms. Use sentence embeddings (they provide code) to map outputs to the nearest valid class. This is copy-paste ready.

2. Control batch size and ordering. They show 2-3% accuracy swings just from changing how many images you feed at once or what order they appear. If you’re comparing models, fix these variables or you’re measuring noise.

3. Reannotate your validation set. Even 10% of your data with multi-label ground truth will reveal if your model is “wrong” or just seeing things the original annotators missed. MLLMs can assist: they show annotators agreed with MLLM predictions 50% of the time on hard cases.

The broader lesson: when models disagree with benchmarks, audit the benchmark before concluding the model is broken. Measurement error compounds faster than model error.