TPIPS: Teaching Vision Models to See Similarity Through Your Lens

Paper: 2607.18237 Authors: Sheng-Yu Wang, Yotam Nitzan, Aaron Hertzmann, Jun-Yan Zhu, Eli Shechtman, Alexei A. Efros, Richard Zhang Categories: cs.CV, cs.LG

The Gap

We’ve had perceptual similarity metrics like LPIPS (Zhang et al., 2018) and its descendants for years now. These metrics take two images and spit out a single number — “how similar are these?” They’ve become the de facto evaluation tool for generative models, image restoration, and retrieval. But here’s the thing everyone quietly knows: that single number is a lossy compression of something fundamentally multidimensional. Two photos of a red apple and a green apple might score as “very similar” on one metric, but that number can’t tell you whether you care about shape, color, texture, or object identity. Human similarity judgments are context-dependent — a painter comparing color palettes makes a completely different assessment than an architect comparing structural forms — yet our metrics pretend this dimensionality doesn’t exist.

Prior attempts to address this have been limited. Some methods use task-specific feature spaces (e.g., style vs. content networks), but these are hand-designed and inflexible. Others try to learn a general embedding, but still collapse to a single vector. The core assumption — that visual similarity is a one-dimensional quantity — is the bottleneck. This paper asks: what if similarity could be conditioned on a natural language description of the aspect you care about?

[Existing metrics: single scalar]
              |
              v
[Collapse multi-dimensional human judgment]
              |
              v
[No way to query specific aspects]
              |
              |
[This paper: collect multi-aspect triplet data]
              |
              v
[Benchmark VLMs --> significant gap vs human consensus]
              |
              v
[Fine-tune VLM with text-conditioned training]
              |
              v
[TPIPS: promptable similarity metric]
              |
              v
[Aligns with human perception + generalizes OOD]
              |
              v
[Unlocks retrieval / search / eval on specific aspects]

The Increment

One sentence: Before this paper, asking “how similar are these two images in terms of texture?” was impossible with a single metric; after this paper, you can ask exactly that in natural language and get a calibrated answer.

Core Mechanism

The paper’s pipeline has two major phases: data collection and model training. First, the authors build a large-scale dataset of human similarity judgments over image triplets. For each triplet (A, B, C), annotators judge whether A is more similar to B or to C — but critically, they do this across multiple free-form semantic aspects. So the same triplet might yield “A is more similar to B in terms of shape” but “A is more similar to C in terms of color.” Each aspect also gets a natural language label written by the annotator, creating a rich, open-ended vocabulary of similarity dimensions.

In the second phase, they take a frontier vision-language model (VLM) and fine-tune it on this data. The VLM receives two images and a text prompt describing the desired aspect (e.g., “texture,” “overall mood,” “structural layout”), and outputs a similarity score. The key architectural choice is that the text prompt is not a post-hoc filter — it’s baked into the representation computation itself, so the model learns to attend to different visual features depending on what the text asks for.

[Image A] [Image B]   [Text Prompt: "color palette"]
     \        |              /
      v       v             v
  [Vision Encoder]  [Text Encoder]
          \              /
           v            v
      [Cross-Attention Fusion]
               |
               v
      [Conditioned Similarity Head]
               |
               v
         [Score: 0.82]

Think of it like a restaurant review platform. Before TPIPS, using LPIPS is like Yelp showing only one overall star rating for each restaurant — it tells you “these two places are about the same quality” but can’t say whether they’re similar in food, ambiance, or price range. TPIPS is like being able to filter by dimension: “compare these two restaurants on food quality” gives you one ranking, while “compare on ambiance” gives you a completely different one. The restaurants (images) don’t change, but the question you ask (text prompt) reshapes what signal the system extracts. The training data — those aspect-annotated triplets — are like the individual user reviews that teach the platform what “food quality” and “ambiance” mean in practice, rather than someone hand-coding rules.

Key Concepts

• Triplet similarity judgments with aspect annotations: Imagine showing someone three photos of dogs — one golden retriever, one yellow lab, one black lab. You ask: “Which two are more similar?” The answer depends on what you’re looking at. If you care about fur color, the golden and the lab are similar. If you care about breed shape, the two labs are more similar. Existing datasets ask this question once and get one answer. This paper asks it many times, once per aspect, and lets annotators name the aspects in their own words. This is the data foundation that makes everything else possible.

• Text-conditioned perceptual metric: Traditional metrics are functions f(A, B) → score. This paper proposes f(A, B, text) → score, where “text” is a free-form description of what dimension of similarity to measure. The conditioning happens inside the model’s architecture — the text prompt modulates which visual features get amplified or suppressed, not just which output head fires. This is fundamentally different from running separate specialized metrics for each aspect.

• Generalization beyond training distribution: The model is trained on specific aspects seen during data collection, but at test time, it can respond to novel text prompts it hasn’t explicitly been trained on. This emergent compositional ability comes from the VLM’s pretrained language understanding — it can infer what “formality of composition” means even if that exact phrase wasn’t in the training data.

Framework Shift

Before (mainstream):                  After (this paper):

 [Img A]---+                          [Img A]---+
           |                                    |
      [Metric f]                           [TPIPS]
           |                            /    |    \
           v                      [text] [text] [text]
       [1 score]                   /       |       \
           |                  [score1] [score2] [score3]
           v                   (shape)  (color)  (mood)
  "These are similar"
   (but in what way??)

From “similarity is a scalar” to “similarity is a function of a question,” the core shift is that the metric now takes a perspective as input rather than pretending perspective doesn’t exist.

Expert Assessment

Problem choice: This is a genuine and well-articulated gap. The single-scalar limitation of LPIPS-style metrics has been a quiet frustration for years — people use them because there’s nothing better, not because they’re satisfied. The rise of VLMs makes this the right time to tackle it; the tools to condition on language now exist in a way they didn’t five years ago. This sits at a natural inflection point in the field’s trajectory.

Method maturity: The fine-tuning approach is pragmatic rather than elegant. There’s nothing architecturally revolutionary here — it’s “take a strong VLM, collect good data, train it well.” But sometimes that’s exactly what’s needed, and the data contribution is substantial. One concern: the reliance on free-form text annotations means the training signal is noisy, and the paper could be more transparent about how they handle annotation disagreements. Are there simpler approaches being overlooked? Possibly a contrastive learning framework with explicit aspect embeddings could be more efficient, but the authors’ choice to stay in natural language space is more flexible and harder to dismiss.

Experimental integrity: The baselines are reasonable — they benchmark zero-shot VLMs and existing metrics, which provides meaningful comparison points. The out-of-distribution generalization tests are a nice touch and add credibility. However, I’d want to scrutinize the human evaluation more carefully: “consensus” among annotators can mask genuine disagreement, and the paper doesn’t spend enough time discussing when human similarity itself is genuinely ambiguous. The numbers look solid, but the evaluation could be more nuanced about edge cases.

Writing quality: The paper is well-written overall, with a clear narrative arc. The related work section is comprehensive without being bloated. Where they cut corners: the discussion of failure modes and limitations is thin. A dedicated section on “when does TPIPS break down?” — with examples of prompt sensitivity, adversarial text inputs, or aspects the model simply can’t capture — would significantly strengthen the paper. The qualitative figures are effective but could use more systematic analysis.

Verdict: weak accept — The dataset alone is a valuable contribution, and the method is practical and well-validated, though the approach is more “competent engineering” than “deep insight,” and the evaluation could probe harder at the boundaries.

Takeaways

• Multi-dimensional evaluation is now practical. If you’re evaluating generative models (diffusion models, GANs, image editors), you can now decompose “perceptual quality” into specific aspects — fidelity to structure, color consistency, texture realism — and report them separately. This is immediately actionable for anyone writing a paper that compares image synthesis methods.

• Text-conditioned metrics are a new design primitive. The idea of a metric that takes a natural language specification of *what to measure is transferable far beyond images. Audio similarity, 3D shape comparison, even document similarity could follow the same pattern: collect aspect-annotated triplets, fine-tune a multimodal model, prompt at inference time.

• The data collection recipe is reusable. The triplet annotation protocol with free-form aspect labels is a template you can adapt for your own domain. The key insight is letting annotators *name the aspects rather than pre-defining them — this captures the long tail of human judgment that rigid taxonomies miss.