Symmetry in Language Statistics Shapes the Geometry of Model Representations

Paper: 2602.15029
Authors: Dhruva Karkada, Daniel J. Korchinski, Andres Nava, Matthieu Wyart, Yasaman Bahri
Categories: cs.LG, cond-mat.dis-nn, cs.CL

Abstract

Neural network representations exhibit remarkable geometric structures that remain theoretically unexplained. This paper establishes a rigorous mathematical connection between statistical symmetries in language data and the geometric organization of learned embeddings. The authors prove that translation symmetry in word co-occurrence statistics—where the probability of two words appearing together depends only on their relative relationship rather than absolute position—directly determines the emergence of simple geometric structures in high-dimensional embedding spaces. Examples include calendar months forming circles, years organizing along smooth one-dimensional manifolds, and geographic coordinates being linearly decodable. Remarkably, these structures persist even under strong perturbations to the training data and at moderate embedding dimensions, a robustness the authors explain through the presence of underlying continuous latent variables controlling co-occurrence patterns.

Key Contributions

• Establishes a mathematical proof linking translation symmetry in co-occurrence statistics to geometric structures in embeddings
• Demonstrates that circular and manifold structures emerge naturally from symmetric statistical patterns in language
• Shows empirically that geometric structures are robust to significant perturbations in training data
• Introduces a theoretical framework explaining robustness through continuous latent variables
• Validates the framework across word embeddings (Word2Vec, GloVe), text embeddings (BERT, RoBERTa), and large language models (GPT-2, Llama)

Theoretical Framework

The core insight revolves around translation symmetry in co-occurrence matrices. For a set of words with an underlying ordering (like months or years), the co-occurrence probability $P(w_i, w_j)$ exhibits translation symmetry when it depends only on $|i - j|$ rather than the specific values of $i$ and $j$. Mathematically:

$P(w_i, w_j) = f(|i - j|)$

The authors prove that when embedding models like Word2Vec or GloVe optimize objectives based on such symmetric co-occurrence statistics, the learned embeddings $\mathbf{v}_i$ naturally organize into geometric structures. For periodic symmetries (like months repeating annually), embeddings form circles. For monotonic relationships (like years or temperatures), they form smooth one-dimensional manifolds.

The key mathematical result shows that the embedding geometry is determined by the spectral properties of the co-occurrence matrix. When this matrix has translation symmetry, its eigenvectors have specific harmonic structures that force the embeddings into predictable geometric configurations. This holds even in high-dimensional spaces where embeddings have hundreds of dimensions, because the relevant structure concentrates in a low-dimensional subspace.

Robustness and Latent Variables

A surprising finding is that geometric structures persist even when co-occurrence statistics are heavily perturbed. The authors demonstrate this by removing all sentences where certain word pairs appear together, yet the circular structure of months remains intact. This robustness cannot be explained by direct co-occurrence alone.

The paper introduces a generative model where co-occurrence statistics are controlled by continuous latent variables. For example, months co-occur not just because of their calendar proximity, but because they share underlying seasonal, cultural, and temporal contexts. These latent variables create a “soft” translation symmetry that is more robust than exact statistical patterns.

Formally, if words are associated with latent variables $\theta_i$ on a continuous space, and co-occurrence probabilities depend on $|\theta_i - \theta_j|$, then even with noisy or incomplete observations, the learned embeddings will approximate the latent space geometry. This explains why removing direct co-occurrences doesn’t destroy the structure—the latent variable relationships persist through indirect statistical dependencies.

Empirical Validation

The authors validate their theory across multiple model architectures and scales:

Word Embeddings: Word2Vec and GloVe trained on Wikipedia and Common Crawl show clear circular structures for months and linear structures for years, consistent with theoretical predictions.

Text Embeddings: Contextual models like BERT and RoBERTa, which generate context-dependent representations, still exhibit these geometric structures in their token embeddings, suggesting the phenomenon is fundamental rather than architecture-specific.

Large Language Models: Even in models like GPT-2 and Llama with billions of parameters, the geometric structures emerge in the residual stream representations. The authors use linear probes to decode latent variables (like month position or geographic coordinates) and find that accuracy remains high even with perturbed training data.

Dimensionality analysis reveals that these structures are robust down to surprisingly low embedding dimensions (around 50-100), though they become clearer in higher dimensions. This suggests the geometric organization is a fundamental feature of how neural networks compress statistical regularities.

Takeaways

1. Geometric structures in neural embeddings are not accidental but mathematically determined by symmetries in the training data’s statistical patterns
2. Translation symmetry in co-occurrence statistics directly predicts whether embeddings will form circles, lines, or other manifolds
3. The robustness of these structures under data perturbation indicates they capture deep statistical regularities mediated by latent variables
4. This framework applies across model architectures from simple word embeddings to large language models with billions of parameters
5. Understanding these geometric principles could inform better embedding design, interpretability methods, and data efficiency in training neural networks