Failure of Contextual Invariance in Gender Inference with Large Language Models

Paper: 2603.23485 Authors: Sagar Kumar, Ariel Flint, Luca Maria Aiello, Andrea Baronchelli Categories: cs.CL, cs.AI, cs.SI

Abstract

This paper tests whether LLM gender judgments stay stable when the task is wrapped in meaning-neutral context. Using a controlled pronoun-choice setup, the authors show that even tiny, theoretically irrelevant context can strongly shift outputs. Patterns tied to gender stereotypes in context-free settings weaken or vanish, while unrelated cues can become unusually influential.

Key Contributions

• Demonstrates that small irrelevant context can cause large changes in gender inference outputs
• Shows stereotype-related signals weaken once context is added
• Identifies cases where an unrelated pronoun cue dominates predictions
• Uses contextuality-by-default analysis to quantify persistent context dependence
• Raises concerns for bias evaluation and high-stakes deployment

Methodology

The study uses a controlled pronoun-choice task to test contextual invariance. The key question is simple: if two prompts are meant to be meaning-equivalent, should the model’s gender judgment stay the same? The authors add small contextual variations that should not matter semantically and then measure how much the model’s answers move.

This setup is useful because it isolates a common evaluation assumption: that a model’s output reflects stable internal beliefs rather than prompt packaging. By probing that assumption directly, the paper exposes how fragile some bias measurements can be.

Results

The main result is that context matters a lot more than expected. Gender judgments that looked stable in a context-free setting shift substantially once irrelevant context is introduced. Stereotype-linked signals become less reliable, and in some cases a cue unrelated to the target referent exerts the strongest influence.

The contextuality-by-default analysis finds persistent context dependence in roughly 19% to 52% of cases, beyond simple repetition effects. That range is wide, but the message is clear: these models are not as context-invariant as many evaluation pipelines assume.

Takeaways

1. Tiny context changes can materially alter LLM gender judgments
2. Bias signals measured in context-free prompts may not generalize to realistic settings
3. Context can suppress stereotype patterns instead of cleanly revealing them
4. A contextuality-by-default lens is helpful for analyzing prompt sensitivity
5. Bias evaluation needs to account for instability under semantically irrelevant context