Embedding Models Measure in Peculiar Ways

Paper: 2609.20821 Authors: Juri Opitz, Andrianos Michail Categories: cs.CL, cs.LG

The Gap

Dense embedding models (BERT, OpenAI text-embeddings, BGE, Cohere) form the backbone of modern information retrieval, retrieval-augmented generation (RAG), and vector databases. Standard benchmarks like MTEB measure subjective semantic similarity (e.g., whether “a dog barking” is close to “a puppy howling”). Because natural language semantics are fuzzy, evaluating whether a vector space’s geometry is genuinely sound remains elusive.

The paper targets a precise, objective testbed: physical measurements. Quantities like mass ($1\text{ kg} = 1000\text{ g}$), distance ($1\text{ m} = 100\text{ cm} \approx 3.28\text{ ft}$), time ($60\text{ s} = 1\text{ min}$), and volume ($1\text{ L} = 1000\text{ mL}$) possess an exact, non-negotiable metric ground truth. Distance in physical reality is continuous, ordered, and unit-invariant.

If text embeddings truly model conceptual semantics rather than lexical surface forms, two expressions representing identical physical quantities should project to identical neighborhoods, and their vector distances should correlate monotonically with ratio-scale physical differences.

   THE TESTBED: PHYSICAL GROUND TRUTH VS EMBEDDING REALITY

   Physical Reality:
     [100 cm] <== identical ==> [1 m]
        |                           |
        | 10x smaller               | 1000x larger
        v                           v
     [10 cm]                     [1 km]
     (Continuous, monotonic, unit-invariant metric space)

                        VS

   Embedding Space Reality:
     "100 cm" <--- closer ---> "100 kg"  (Shared token "100")
        |
        +-- far from -- "1 m"           (Surface string mismatch)
        |
        +-- chaotic order vs "10 cm"    (Broken ordinal geometry)

The Increment

One sentence: Dense text embeddings only weakly encode objective physical measurements, exhibiting peculiar geometric distortions where semantic proximity is governed by superficial digit and string overlap rather than real-world magnitude or unit equivalence, and recalibration fails to resolve the misalignment.

Core Mechanism

The authors construct systematic measurement suites across four fundamental physical dimensions: mass, distance, time, and volume. Within each dimension, they evaluate:

1. Unit Equivalence: Can the model recognize that $1000\text{ mg}$ equals $1\text{ g}$?
2. Metric Distance Correlation: Does cosine distance or Euclidean distance between $x$ and $y$ track $\log |x - y|$ or relative ratio differences?
3. Cross-dimensional Intrusion: Does $100\text{ meters}$ attract $100\text{ grams}$ more strongly than $0.1\text{ kilometers}$ due to token sharing?

The findings uncover three distinct pathology patterns:

• Digit Anchoring: The numeral token exerts an overwhelming gravitational pull. Two completely unrelated physical dimensions sharing the same number string (e.g. “100 seconds” and “100 liters”) routinely sit closer in vector space than identical values across unit conversions (“1 minute” and “60 seconds”).
• Unit Clustering over Magnitude: Vectors cluster by surface unit names rather than scale. Millimeters sit near centimeters regardless of whether the scale is nanoscopic or planetary.
• Invariance Breakdown: Linearly projecting the embedding space or fitting calibration parameters (affine transformations, temperature scaling) yields negligible improvement. The defect is topological, not a simple global scale mismatch.

   PATHOLOGY OF EMBEDDING MEASUREMENT

   [ Physical Query: "500 grams" ]
                 |
                 +--> High Similarity: "500 kilometers"  [Shared prefix "500"]
                 +--> Moderate:        "100 grams"       [Shared unit word]
                 +--> Low / Distant:   "0.5 kilograms"   [Actual Physical Equivalent!]

The structural metaphor is a librarian organizing an encyclopedia by font size and first letters instead of topic. Imagine asking for a map of the solar system. The librarian hands you a book on plumbing because both book titles happen to have the number “9” printed in bold red ink. The embeddings act not as a physical measuring tape, but as a visual string pattern matcher wearing a semantic badge. It recognizes the typographical ink of the digits, but has no sensory or causal grounding in what mass actually feels like or how distance actually unfolds in space.

Key Concepts

• Physical Ground Truth as a Probe: Unlike emotional or abstract language where “similarity” is inherently subjective, physical measurements provide unambiguous mathematical ground truth ($d_{\text{physical}}(A, B)$ is exact).
• Surface Form Bias in Dense Vectors: The tendency of contrastive embedding training (e.g., InfoNCE on text pairs) to exploit token overlap as an easy contrastive shortcut, failing to learn deep functional abstractions.
• Topological Incoherence: When relational geometry cannot be repaired via linear transformations because the neighborhood graphs themselves are folded along token artifacts rather than scale dimensions.

Framework Shift

Before (Assumed Semantic Grounding):
  Dense Embeddings grasp real-world concepts
  -> "1000 meters" and "1 kilometer" share the same semantic core
  -> Distance in vector space reflects conceptual proximity

After (Observed Lexical Surface Shortcut):
  Dense Embeddings rely heavily on superficial string & digit co-occurrence
  -> "1000 meters" is pulled toward "1000 dollars" by token co-occurrence
  -> Physical equivalence is systematically shattered across unit conversions
  -> Vector search on numerical and dimensional data requires explicit symbolic parsing

From “trusting dense embeddings to navigate physical quantities,” the core shift is recognizing that embedding spaces lack dimensional grounding, requiring symbolic normalization before vector ingestion.

Expert Assessment

Problem choice: Refreshing and conceptually incisive. Much of NLP is saturated with incremental benchmarks where ambiguous labels obscure model failures. Testing embeddings against unambiguous physical laws provides a clean, rigorous lens on representation geometry.

Method maturity: The experimental design is elegant. Isolating mass, distance, time, and volume across standard prefixes and non-metric conversions tests the limits of contrastive learning. Attempting post-hoc recalibration before concluding that the flaw is structural demonstrates commendable scientific hygiene.

Experimental integrity: Tested across multiple leading embedding architectures (both proprietary APIs and open-weight models). The diagnostic probes are reproducible and avoid cherry-picking by sweeping whole numeric sequences across multiple decades of magnitude.

Writing quality: Lucid, concise, and focused. The title (“Embedding Models Measure in Peculiar Ways”) is matched by sober, measured claims that resist over-generalizing beyond measured physical spaces.

Verdict: strong accept — An essential reality check for vector search practitioners and a foundational diagnostic on the limits of pure text-contrastive representation learning.

Takeaways

• Never rely on raw vector embeddings for semantic search over physical, engineering, or numerical quantities (e.g. searching “parts under 5mm” or “deliveries within 2 miles”).
• Always parse physical units and numerical scales into structured symbolic metadata (e.g. normalized SI units in a SQL/metadata filter) before vector matching.
• Contrastive text representation learning requires explicit relational inductive biases if it is ever to support scientific and embodied reasoning.