GeoFidelity-Bench: Do Text-to-Image Street-View Models Actually Know Which Road They’re Drawing?

Paper: 2606.23669 Authors: Kaizhen Tan, Hanzhe Hong, Siru Tao Categories: cs.CV

The Gap

Existing text-to-image models (Stable Diffusion, FLUX, SDXL, etc.) routinely churn out city street scenes that look “right” — correct building styles, road markings, and even local vegetation. But “looking right” for a generic city is not the same as “looking like this specific road segment”. Prior evaluation benchmarks (e.g., Cityscapes, Mapillary-based FID, or even GPS-conditioned generation papers) either measure global image quality or test coarse geographic attributes (city-level). They never ask: if I prompt for “Main Street, Springfield”, does the output actually resemble the real Main Street in Springfield, rather than some other street in Springfield?

This paper identifies that gap explicitly. They design a reference-panel ranking setup where the test is not absolute similarity to the target, but whether the target ranks above all other segments. This forces the model to produce segment-discriminating features, not just city-style plausibility.

[Problem: Models generate plausible city streets]
         |
         v
[Assumption: Existing metrics (FID, IS) measure global quality, not segment fidelity]
         |
         v
[Method: Reference-panel benchmark with ranking (target vs nearest segment vs other city segments)]
         |
         v
[Evidence: Adding street/neighborhood names boosts top-1 accuracy by +5.5pp, but margin to nearest segment is ~0]
         |
         v
[Conclusion: City-plausible generation exists without segment-level fidelity; local names help plausibility more than identity]

The Increment

One sentence: Before this paper, we could tell if a generated street looked like a city; after this paper, we can tell if it looks like a *specific road within that city — and the answer is largely no.

Core Mechanism

The benchmark works in three layers.

First, reference construction. They pick 109 named OpenStreetMap road segments across 25 cities (6 continents). For each segment, they collect 5–10 real Mapillary images (panels) that fall *on that segment. This creates a per-segment “photo album” — multiple viewpoints of the same road.

Second, generation and querying. They take a text-to-image model and generate a street-view image conditioned on a prompt (e.g., “a street view of University Avenue in Palo Alto”). That generated image becomes the query. For each query, they compute image-level features (using a pre-trained backbone like DINOv2) and compare to all reference panels from all segments. The result is a ranked list: which real panel is most similar to the generated image?

Third, evaluation. The key metric is Top-1 Retrieval Accuracy: does the top-ranked panel come from the *target segment (the one named in the prompt)? They also compute the similarity gap between the target panel and the panel from the nearest other segment in the same city. This gap tells whether the model is just “close enough” (small gap) or truly distinguishing (large gap).

[Reference: Segment S_i with panels P_i1 ... P_ik]
         |
         v [DINOv2 feature extractor]
[Feature vectors for all reference panels]
[Generated image] --[DINOv2]--> [query feature vector]
         |
         v
[Cosine similarity: query vs all panel features]
         |
         v
[Rank panels; check if any panel from target segment is top-1]
[Evaluate: Top-1 retrieval accuracy]
[Evaluate: similarity margin between target and nearest same-city segment]

Structural metaphor: Think of each road segment as a person, and its reference panels as different photos of that person (from different angles, times of day). The generation model is a police sketch artist who gets a description: “Draw Mr. Smith from 123 Oak Street.” The artist produces a sketch. Now you take that sketch to a photo lineup containing Mr. Smith’s photos, photos of his next-door neighbor (nearest segment), photos of other people in the same city, and photos from other cities. The “lineup” is the ranking. The central question: does the artist’s sketch make the police pick Mr. Smith’s photo first? The paper finds that when you give the artist a detailed description (“Mr. Smith, 123 Oak Street, Springfield”), the sketch becomes more likely to be recognized as *someone from Springfield, but rarely as Mr. Smith specifically. The margin between Mr. Smith and his neighbor is tiny — the sketch could be either. That is the gap.

Key Concepts

• Reference-panel superiority: Instead of comparing a generated image to a single “ground truth” image (which is ill-posed because a street looks different at different times), the benchmark uses multiple reference images per segment. The model must produce an image that is *most similar to one of the target’s references among all references. This converts a regression problem into a ranking problem, which is more robust and directly measures discriminability.

• Local plausibility vs. segment identity: A generated image can be “locally plausible” — correct sidewalks, trees, building styles — while being visually identical to dozens of other streets in the same neighborhood. The paper’s key insight is that the similarity margin between the target and the nearest other segment quantifies the *distinctiveness of the model’s output. A margin near zero means the model fails to encode segment-specific details (e.g., a unique intersection geometry, a specific storefront sign).

Framework Shift

Before (mainstream approach):        After (this paper):
[Prompt: "Street view in Palo Alto"] [Prompt: "Street view in Palo Alto"]
      |                                     |
      v                                     v
[Generate image]                      [Generate image]
      |                                     |
      v                                     v
[Compare to single reference (if any)] [Compare to panel set; rank across all segments]
      |                                     |
      v                                     v
[Score: FID / LPIPS / human eval]     [Score: Top-1 retrieval acc + margin to nearest]

One sentence: From single-image absolute similarity to multi-segment ranking for discriminability, the core shift is that the metric now demands the model to produce segment-specific features rather than mere city-style plausibility.

Expert Assessment

Problem choice: Real gap. The paper correctly identifies that existing metrics are blind to geographic specificity, and the benchmark is elegantly designed to expose this. It sits at the intersection of geographic grounding and generative evaluation — a niche that will grow. Not manufactured; the controlled experiments with wrong names confirm the gap is there.

Method maturity: Clever insight with moderate engineering. The reference-panel ranking is a natural adaptation of image retrieval evaluation. The authors could have used simpler baselines (e.g., CLIP retrieval) but chose DINOv2, which is reasonable. The main novelty is not in the algorithm but in the problem framing and dataset construction. Some might argue that Gaussian blurs or synthetic negatives could simplify, but that would miss the point: the reference panels are real and contain the exact segment-level variation the model must match.

Experimental integrity: Baselines are fair (city-only vs. street+neighborhood vs. GPS). The confidence intervals are given and properly computed. One red flag: the “nearest segment in the same city” is defined by geographic distance, not by perceptual similarity — two physically adjacent roads might look very different (e.g., a highway vs. a residential lane), and the margin could be artifically large. The authors acknowledge this but could have used perceptual ground-truth clusters. Also, the number of models (six) is decent but all open-weight; closed models like DALL·E 3 are missing. The GPS-as-text experiment is a bit naïve — it’s known that raw GPS coordinates are poorly handled by text encoders; a negative result is not surprising.

Writing quality: The paper is clear and well-structured, but the “Related Work” section is thin — it barely mentions geographic grounding literature (e.g., OSMP, geotagged training). The most missing piece is a failure analysis section: when does the model succeed? Are there segments with distinctive landmarks (famous buildings, bridges) that get high accuracy? Without that, we don’t know if the benchmark is unfairly hard or genuinely informative. A revision that adds a qualitative breakdown by segment type would elevate the paper significantly.

Verdict: weak accept — the problem is important, the benchmark is solidly built, and the core finding is non-trivial, but the lack of failure analysis and the limited model scope prevent it from being a definitive reference.

Takeaways

• Ranking over regression: When your ground truth is ambiguous (multiple valid outputs), use retrieval-based metrics rather than pixel-wise or perceptual distances. This trick transfers to any “generate and compare to a reference set” scenario (e.g., 3D scene generation, facial generation).
• Margin to nearest negative: The gap between the target and the closest non-target is more informative than Top-1 accuracy alone. Always report this margin — it separates “barely correct” from “clearly distinct”.
• Prompt engineering for geographic specificity: The paper shows that adding street and neighborhood names boosts performance by ~5.5 percentage points, but the gain is fragile (wrong names still help). Practitioners should be warned that their model may be latching on to generic local features (e.g., “Palo Alto = palm trees and low-rise buildings”) rather than true segment identity.