Fixing Vision Transformers' Hidden Bias: Why Your Model Cares Where Things Are (Even When It Shouldn't)

Paper: 2603.16840 Authors: Moritz Pawlowsky, Antonis Vamvakeros, Alexander Weiss, Anja Bielefeld, Samuel J. Cooper, Ronan Docherty Categories: cs.CV, cond-mat.mtrl-sci

The Gap

Vision transformers like DINOv2 are supposed to learn “what” things are, not “where” they are. But they secretly encode position information into their features. This is fine for natural images (a sky is usually at the top), but breaks down for material science microscopy where a grain boundary should look the same whether it’s in the corner or center of the image. Prior work has documented positional biases in language models and some vision models, but nobody has systematically measured it across different ViT architectures or shown how to remove it without destroying the model’s semantic understanding.

Problem: ViTs encode position into features
    |
    v
Assumption: Positional encoding choice drives this bias
    |
    v
Method: Replace absolute/learned PE with ALiBi (relative)
    |
    v
Evidence: Linear probes show reduced position prediction
          + Segmentation tasks maintain performance
    |
    v
Conclusion: ALiBi removes positional bias while preserving semantics

The Increment

One sentence: Before this paper, you couldn’t use foundation vision models for zero-shot tasks on rotationally symmetric data (like microscopy) because they leaked position information; after, you can finetune them with ALiBi to get unbiased features that still understand semantics.

Core Mechanism

The method has three stages. First, they measure positional bias by training linear probes on frozen ViT features to predict patch positions. If the probe achieves high accuracy, the features contain positional information. They test this across DINOv2, MAE, and CLIP with different positional encodings (absolute sinusoidal, learned, rotary).

Second, they finetune these models by replacing their positional encoding with ALiBi (Attention with Linear Biases). ALiBi doesn’t add position embeddings to tokens. Instead, it adds a bias directly to the attention scores based on the distance between query and key positions. The bias is negative and grows linearly with distance, so nearby patches attend more to each other.

Third, they validate that the finetuned models retain semantic understanding by testing on segmentation tasks. They use the unbiased features as input to trainable segmentation heads on microscopy images of battery materials and porous media.

Input Image
    |
    v
[Patch Embedding] ---> [ViT Blocks with ALiBi]
    |                        |
    |                   (attention bias = -m * |i-j|)
    |                        |
    v                        v
Features -----> [Linear Probe] ---> Position Prediction
    |                                (should fail if unbiased)
    |
    v
[Segmentation Head] ---> Semantic Output
                         (should succeed if semantics preserved)

Think of it like a restaurant critic who unconsciously rates corner tables higher than center tables. You want them to judge food quality, not seating location. The original ViT is like giving each table a name tag (“Table 7, northwest corner”) that the critic sees. Even if they try to ignore it, the information leaks into their review. ALiBi is like replacing name tags with a simple rule: “Pay slightly more attention to nearby tables when comparing dishes.” The critic still understands food quality (semantics) but can’t tell you which table a dish came from (position).

Key Concepts

• Positional Encoding: Vision transformers chop images into patches and process them as a sequence. But unlike text, image patches don’t have inherent order. Positional encoding adds information about where each patch came from. Absolute encodings (like sinusoidal or learned vectors) give each position a unique signature. Relative encodings (like ALiBi) only encode the distance between positions. Imagine a classroom where absolute encoding gives each desk a unique ID number, while relative encoding just tells you “this desk is 2 rows behind that one.” For tasks where absolute position shouldn’t matter (like identifying a cell type in microscopy), absolute encodings leak information that biases the model.

• Linear Probing: A diagnostic tool to measure what information is encoded in learned features. You freeze the model’s weights and train only a simple linear classifier on top. If the classifier succeeds at a task, the frozen features must contain the necessary information. It’s like testing if a locked diary contains secrets by seeing if you can guess the secrets just from the diary’s weight and texture—without opening it. In this paper, they probe whether features encode position by training a linear layer to predict patch coordinates. High accuracy means position leaked into the features.

• ALiBi (Attention with Linear Biases): A relative positional encoding that modifies attention scores instead of token embeddings. In standard attention, query Q and key K produce a score Q·K. ALiBi adds a penalty: Q·K - m·|i-j|, where i and j are positions and m is a learned slope. Nearby positions get small penalties, distant ones get large penalties. This creates a locality bias without encoding absolute position. It’s like judging a neighborhood by saying “I prefer houses close to mine” rather than memorizing every address. The model learns spatial relationships (this patch is near that one) without learning coordinates (this patch is at row 5, column 3).

Framework Shift

Before (mainstream approach):        After (this paper):

Image --> [Patch + Pos Embed]       Image --> [Patch Embed]
              |                                  |
              v                                  v
         [Transformer]                  [Transformer + ALiBi]
              |                                  |
         (features encode                   (attention biased
          absolute position)                 by relative distance)
              |                                  |
              v                                  v
         Downstream Task                   Downstream Task
         (biased by position)              (unbiased features)

From baking position into the features to baking it into the attention mechanism, the core shift is moving from “where am I?” to “how far apart are we?”

Expert Assessment

Problem choice: Real gap, not manufactured. Material scientists actually face this—microstructure images have no preferred orientation, but foundation models trained on natural images assume they do. The problem sits at the intersection of domain adaptation and architectural choices, which is underexplored territory.

Method maturity: Clever insight, not brute force. ALiBi already existed for language models; applying it to vision and measuring the bias reduction is straightforward but effective. However, they don’t explore why ALiBi specifically works better than other relative encodings (like rotary), or whether simpler solutions (like data augmentation with random rotations) could achieve similar results.

Experimental integrity: Baselines are fair. They test multiple architectures (DINOv2, MAE, CLIP) and positional encodings. The linear probing methodology is sound. One concern: they only test on microscopy images for downstream tasks. Would be stronger to show the unbiased features don’t hurt performance on natural images too. The segmentation results are qualitative (visual comparisons) rather than quantitative metrics, which weakens the claim about preserved semantics.

Writing quality: The paper is clear but front-loads too much motivation about material science. A general computer vision reader might bounce off the intro. The methods section is solid. The results would benefit from ablations—what happens with partial finetuning? How much data is needed? The discussion of why ALiBi works is thin; they could dig deeper into the attention patterns.

Verdict: weak accept — Addresses a real problem with a simple, effective solution, but experimental validation is narrow and lacks depth on why it works.

Takeaways

If you’re adapting foundation models to domains where position shouldn’t matter (medical imaging, satellite imagery, scientific microscopy), try ALiBi positional encoding. The finetuning is cheap and the bias reduction is measurable via linear probing—use that as a diagnostic before deploying. More broadly, this paper shows that architectural choices in foundation models can encode implicit assumptions about data structure. When your domain violates those assumptions, you can often fix it by swapping components (like positional encoding) rather than retraining from scratch. The linear probing technique is also worth stealing: it’s a quick way to audit what information your features contain before you commit to a downstream task.