SOCO: Testing Whether Vision Models Really Understand Object Parts

Paper: 2605.31597 Authors: Olaf Dünkel, Basavaraj Sunagad, Haoran Wang, David T. Hoffmann, Christian Theobalt, Adam Kortylewski Categories: cs.CV

The Gap

Vision foundation models claim to understand objects, but we lack a consistent way to test whether they grasp part-level structure. Existing semantic correspondence benchmarks suffer from inconsistent annotations, limited category coverage, and no way to evaluate vision-language models on fine-grained part understanding. We can measure ImageNet accuracy, but that doesn’t tell us if a model knows that a dog’s ear corresponds to a cat’s ear, or whether it can locate “the left front wheel” when asked in natural language.

Problem: How to measure structured object understanding?
   |
   v
Gap: Existing SC benchmarks have inconsistent protocols,
     limited categories, no language grounding
   |
   v
Assumption: Part-level correspondence ability reflects
            deeper semantic understanding
   |
   v
Method: SOCO benchmark with taxonomy of correspondence types,
        1M+ pairs across 100 categories, language descriptions
   |
   v
Evidence: (1) Models encode structure but fail cross-category transfer
          (2) LVLMs better at language prompts than visual matching
          (3) SC performance predicts downstream tasks > ImageNet
   |
   v
Conclusion: Correspondence is a better probe of representation
            quality than classification accuracy

The Increment

One sentence: Before SOCO, we evaluated vision models on classification; after SOCO, we can systematically test whether they understand that functionally similar parts correspond across objects, even when those objects look completely different.

Core Mechanism

SOCO introduces a three-layer evaluation framework. First, it defines a taxonomy of correspondence types: within-instance (same object, different views), within-category (different instances of same class), and cross-category (functionally similar parts across different classes). Second, it provides dense keypoint annotations for 100 object categories with over 1 million correspondence pairs, where each keypoint has a functional semantic label like “left_front_wheel” or “beak_tip”. Third, it includes natural language descriptions for each keypoint, enabling evaluation of vision-language models through text-prompted localization tasks.

The evaluation protocol tests models in two modes. For vision-only models, given a keypoint in a source image, the model must locate the corresponding point in a target image—potentially of a different category. For vision-language models, given a text description like “the animal’s left ear”, the model must ground that description to pixel coordinates. The benchmark measures performance across the correspondence taxonomy, revealing where models succeed (within-category matching) and where they fail (cross-category transfer).

Input Pipeline:
  Source Image + Keypoint  ------>  Vision Encoder
       |                                  |
       v                                  v
  Target Image  ------------>  Feature Matching  ---> Predicted Keypoint
                                      |
  Text Description -------->  Language Grounding --> Predicted Location
                                      |
                                      v
                            Evaluate across taxonomy:
                            - Within-instance
                            - Within-category  
                            - Cross-category

Think of SOCO as a language proficiency test, but for visual understanding. Just as we don’t test language ability by asking “can you read?”, we shouldn’t test vision models by asking “can you classify cats?” A real language test checks whether you understand that “perro” and “dog” refer to the same concept, whether you can use pronouns correctly across sentences, and whether you grasp metaphorical language. SOCO does the same for vision: it checks whether a model understands that a bicycle’s handlebar and a motorcycle’s handlebar serve the same function (cross-category correspondence), whether it can track a specific wheel across different viewing angles (within-instance), and whether it knows that “left front wheel” means something consistent across vehicles (language grounding). The taxonomy is like test sections (grammar, vocabulary, comprehension), the 1M pairs are like test questions, and the language descriptions are like the translation section.

Key Concepts

• Semantic Correspondence: Imagine you’re looking at two photos of different dogs—one a golden retriever lying down, one a poodle standing up. Semantic correspondence is the ability to point at the retriever’s left ear and correctly identify where the poodle’s left ear is, despite differences in breed, pose, and appearance. It’s not about pixel similarity (the ears look nothing alike) but functional equivalence (both are left ears). This differs from feature matching in traditional computer vision, which finds visually similar patches. A model with good semantic correspondence understands object structure at a conceptual level—it knows what parts are, not just what they look like.

• Correspondence Taxonomy: Not all matching tasks are equally hard. Matching a car’s wheel across two photos of the same car from different angles (within-instance) is easier than matching wheels across a sedan and a truck (within-category), which is easier than matching a car’s wheel to a bicycle’s wheel (cross-category). The taxonomy organizes these difficulty levels systematically. It’s like the difference between recognizing your friend’s face in different lighting (easy), recognizing a stranger who looks similar to your friend (medium), and recognizing that both humans and cartoon characters have “eyes” in analogous positions (hard). The taxonomy reveals which levels of abstraction a model has actually learned.

• Language-Grounded Localization: This tests whether a model can connect text descriptions to visual locations. Given “the left front wheel” as text, can the model point to the correct wheel in an image? This is harder than it sounds because it requires understanding spatial relations (left vs right), part hierarchies (wheel is part of vehicle), and functional roles (front wheels steer). It’s the difference between a model that has memorized “wheel = round thing at bottom” versus one that understands vehicle structure. For vision-language models, this reveals whether their language grounding is truly fine-grained or just coarse object-level associations.

Framework Shift

Before (mainstream approach):        After (this paper):

Evaluate on ImageNet:                Evaluate on correspondence:
                                     
  Image --> Classifier --> Label       Source + Target --> Matcher
     |                                      |
     v                                      v
  "cat" or "dog"?                    "Where is the corresponding
                                      part in target image?"
                                     
  Measures: Top-1 accuracy            Measures: PCK across taxonomy
  Scope: Object-level recognition     Scope: Part-level structure
  Signal: Can distinguish classes     Signal: Understands part relations

From classification to correspondence, the core shift is from asking “what is this?” to “where is the equivalent part?”—moving evaluation from category labels to structural understanding.

Expert Assessment

Problem choice: This is a real gap. The field has been using ImageNet accuracy as a proxy for visual understanding for over a decade, but recent work shows models can ace ImageNet while failing at basic spatial reasoning. Semantic correspondence directly tests part-level structure, which matters for robotics, 3D reconstruction, and any task requiring geometric understanding. The timing is right—foundation models are mature enough that we need better diagnostics than classification.

Method maturity: The benchmark design is solid but not groundbreaking. Dense keypoint annotation across 100 categories is labor-intensive but straightforward. The taxonomy (within/cross instance/category) is intuitive and well-motivated. The language descriptions are a nice touch for evaluating LVLMs, though the paper doesn’t deeply explore how language grounding interacts with visual correspondence. The evaluation protocol is clean: percentage of correct keypoints (PCK) at multiple thresholds. No methodological red flags, but also no surprising technical insights—this is careful benchmark engineering, not algorithmic innovation.

Experimental integrity: The experiments are comprehensive and the findings are credible. Testing multiple vision backbones (ResNet, ViT, DINOv2) and LVLMs (GPT-4V, Gemini) with consistent protocols is good practice. The result that correspondence predicts downstream performance better than ImageNet is the paper’s strongest contribution, validated across segmentation, tracking, pose estimation, and 3D detection. However, the paper could be more critical about what “predicts downstream performance” actually means—correlation doesn’t tell us whether improving correspondence would improve those tasks, or whether both are driven by a third factor like model capacity.

Writing quality: The paper is well-structured but verbose in places. The introduction spends too much time motivating semantic correspondence (the CV community already knows why it matters) and not enough explaining what’s novel about SOCO versus prior benchmarks like SPair-71k. Section 3 (benchmark construction) could be condensed—readers don’t need paragraph-long descriptions of annotation procedures. The experimental section (Section 4) is where the paper shines, but even there, some results are presented without sufficient analysis. For example, why do LVLMs fail at cross-image matching but succeed at text-prompted localization? The paper notes this gap but doesn’t dig into mechanisms.

Verdict: weak accept — A useful benchmark that will likely see adoption, with solid experimental validation of correspondence as a better probe than classification, but limited conceptual novelty and some missed opportunities for deeper analysis.

Takeaways

For benchmark designers: The taxonomy of correspondence types (within/cross instance/category) is a clean way to organize evaluation difficulty. Steal this structure for other structured prediction tasks—it’s more informative than a single aggregate metric.

For model developers: If you’re building vision backbones, test on semantic correspondence early. The paper shows that models can have strong within-category performance but collapse on cross-category transfer, suggesting they’re memorizing category-specific part configurations rather than learning general part concepts. This is a concrete failure mode to target.

For practitioners: Don’t trust ImageNet accuracy as a proxy for downstream performance. The paper’s correlation analysis shows correspondence ability predicts task performance more reliably. If you’re choosing between pretrained models, run a quick correspondence eval on your domain—it’s more predictive than classification benchmarks.

For LVLM researchers: The gap between text-prompted localization (where LVLMs do well) and visual-reference matching (where they fail) suggests these models are better at language-to-vision grounding than vision-to-vision reasoning. This is actionable: if your task involves cross-image reasoning, don’t assume LVLM performance on language-grounded tasks will transfer.