Your LLM Knows What It Doesn't Know — But Won't Tell You

Paper: 2607.07670 Authors: Grzegorz Brzezinka Categories: cs.CL, cs.LG

The Gap

People have been trying to detect LLM hallucinations for years. The main approaches fall into two camps: output-level methods (confidence scores, semantic entropy, self-consistency) that require multiple forward passes and cost 5x+ inference time, and representation-level probes (linear classifiers, probing tasks) that look at internal activations. Both camps have a blind spot.

Output-level methods conflate two distinct questions: “Have I seen this entity before?” and “Do I actually know the correct answer?” They treat all uncertainty the same way. Representation-level probes, meanwhile, have mostly been studied on English, with hand-picked entity lists, and without cleanly separating familiarity from correctness across model scales.

The specific gap: nobody has asked whether a model’s internal activations encode entity familiarity before a single token is generated, whether that signal is already strong in small models, and whether it scales differently from factual reliability. If these are distinct phenomena on different curves, that changes how we think about hallucination mitigation — you can’t just scale your way to trustworthiness.

Problem: LLMs hallucinate about entities they haven't seen
         |
         v
Assumption: internal activations should reflect familiarity
            before generation begins
         |
         v
Method: probe post-SwiGLU MLP activations with
        unsupervised dispersion measures (IPR, spectral entropy)
        + supervised linear probe, across 4 model scales
         |
         v
Evidence: AUROC 0.95-1.00 separating known vs fabricated
          BUT familiarity signal at ceiling at 1.5B
          while factual accuracy scales 0/42 -> 19/42
          AND models almost never abstain (2 refusals / 2520 answers)
         |
         v
Conclusion: entity familiarity and factual reliability
            are distinct phenomena on different scaling curves;
            the model "knows what it doesn't know" but won't say so

The Increment

One sentence: Before this paper, we assumed better models meant fewer hallucinations; after this paper, we know the internal “I’ve seen this before” signal is already perfect at the smallest scale — the bottleneck is behavioral, not representational.

Core Mechanism

The method is refreshingly simple. Take any prompt about an entity. Run it through the model for a single forward pass, but stop before generating any output. Extract the activations from the MLP layers — specifically the post-SwiGLU activations, which are the hidden states right after the gated activation function in each transformer block.

From these activations, compute two unsupervised dispersion metrics. Inverse Participation Ratio (IPR) measures how “spread out” the activation vector is — a high IPR means the activations are diffuse (many dimensions contributing), while a low IPR means they’re concentrated on a few dimensions. Spectral Entropy does something similar but from an information-theoretic angle: it measures the entropy of the eigenvalue spectrum of the activation matrix. Both capture the same underlying intuition: when the model encounters a familiar entity, its activations organize into a distinctive, structured pattern; when it sees a fabricated name, the activations are more uniform or scattered.

For comparison, they also train a simple supervised linear probe on top of the same activations. The unsupervised measures hit AUROC 0.95-1.00; the supervised probe hits 0.99-1.00. The key validation: permutation tests confirm this isn’t luck (floors at 0.70-0.74), and the signal holds across held-out layers (0.93-0.99), transfers across entity types (athletes, cities, writers, musicians), and persists even when comparing known vs. obscure-but-real entities (0.96-1.00).

Single forward pass (no generation)
         |
         v
Extract post-SwiGLU MLP activations per layer
         |
    +----+----+
    |         |
    v         v
 [Unsupervised]    [Supervised]
 IPR / Spec. Ent.  Linear Probe
    |                   |
    v                   v
 Dispersion score   Learned classifier
 (no labels needed)  (labels needed)
    |                   |
    v                   v
 AUROC 0.95-1.00    AUROC 0.99-1.00
 separating           separating
 known vs fabricated  known vs fabricated

Think of it like this. Imagine walking into a library and asking the librarian about a book. If the librarian immediately reaches for a specific shelf with confident, precise movements, she’s probably read it. If she freezes, looks around vaguely, or starts scanning shelves randomly, she probably hasn’t. The dispersion measures are watching the “librarian’s movements” — the shape of the model’s internal activity pattern — without ever waiting to hear what she says. The brilliant part: the librarian’s body language (activations) is already perfectly informative at age 5 (1.5B parameters), but her ability to actually tell you accurate facts about the book (factual reliability) improves dramatically as she grows up to age 20 (11B parameters). And here’s the kicker: even when the librarian clearly doesn’t know the book, she never says “I haven’t read that one” — she just confidently makes something up.

Key Concepts

• Inverse Participation Ratio (IPR): Imagine you have 4096 buckets and you’re throwing sand into them. If all the sand lands in 3 buckets, the IPR is low — the activity is concentrated. If the sand spreads evenly across all 4096 buckets, the IPR is high — the activity is diffuse. When the model sees “Adam Małysz” (famous Polish ski jumper it’s seen in training), its activation sand concentrates into specific buckets in a recognizable pattern. When it sees “Zbigniew Kowalczyk-Paprotnik” (a fabricated name), the sand scatters in a generic, unstructured way. The IPR number captures this difference in a single scalar.

• Activation Dispersion vs. Behavioral Reliability: These are the two faces of the paper’s core insight. “Dispersion” is what happens inside the model’s neurons — the geometric shape of activation patterns. “Behavioral reliability” is what comes out of the model’s mouth — correct, verifiable answers. The paper shows these two things have fundamentally different scaling curves. It’s like discovering that a student’s ability to *recognize exam questions (familiarity) maxes out in elementary school, while their ability to answer them correctly (reliability) keeps improving through college. The recognition signal is already perfect; the knowledge is what scales.

• Abstention Gap: The model has an internal signal that says “I don’t know this” (unfamiliar entity = scattered activations), yet it almost never uses that signal to say “I don’t know” in its output. Out of 2,520 answers audited, only 2 refusals and 1 hedge were found. This is the paper’s most provocative finding: the bottleneck in hallucination isn’t the model’s awareness of its own ignorance — it’s the model’s willingness (or architectural inability) to act on that awareness.

Framework Shift

Before (mainstream approach):          After (this paper):

  Model outputs answer                 Look at activations BEFORE
        |                              any answer is generated
        v                                    |
  Check confidence /                       v
  run N samples for                   Compute single-pass
  self-consistency                    dispersion metrics
        |                                    |
        v                                    v
  Expensive (5x+ inference)           Cheap (1 forward pass, no generation)
  Conflates familiarity               Separates familiarity from
  and correctness                     correctness as distinct signals
  Treats all uncertainty equally      Shows they scale on different curves

From “detect hallucinations by examining what the model says” to “detect what the model knows by examining how its neurons move before it speaks,” the core shift is relocating the diagnostic moment from output to representation.

Expert Assessment

Problem choice: This is a genuine and underexplored gap. The distinction between entity familiarity and factual reliability has been implicit in prior work on calibration and hallucination, but nobody has made it this clean and this systematic. The focus on Polish-language models (Bielik) is both a strength — it tests the signal in a lower-resource setting where training data is more constrained — and a limitation, since we don’t know if the findings transfer to English or Chinese models. The entity taxonomy (known/obscure-but-real/fabricated) is well-designed and the 42+42+42 split across 4 domains gives reasonable coverage.

Method maturity: The unsupervised dispersion measures are clever and elegant — IPR and spectral entropy are well-known in physics and information theory, but applying them to post-SwiGLU activations as a familiarity proxy is novel. The supervised linear probe serves as a useful ceiling comparison but doesn’t add much conceptual novelty. The matched-template counterfactual (varying templates to rule out template-driven artifacts) shows good experimental hygiene. One concern: the paper doesn’t compare against other representation-level probes like CKA or representation similarity analysis, which might reveal more about *what the dispersion signal actually encodes.

Experimental integrity: The numbers are strong and the validation is thorough — permutation tests, held-out layers, cross-entity-type transfer, template ablations. The 5x-sample semantic entropy baseline at 0.71-0.83 vs. the paper’s 0.95-1.00 is a convincing cost-accuracy argument. However, the “within known entities” result (separating correct from hallucinated answers) is much weaker — probe AUROC 0.93 is decent but the unsupervised measures don’t beat a first-token-entropy baseline. The paper is honest about this, which is good. The refusal audit (2/2520) is a striking data point but it’s not clear whether this is Bielik-specific behavior or a general LLM trait.

Writing quality: The paper is well-structured and the narrative arc is clear. The related work section could be tighter — it’s somewhat scattered between hallucination detection, calibration, probing, and scaling laws without a clean synthesis. The entity lists (42 per domain) are somewhat arbitrary; a brief justification for why 42 and not 100 would help. Section 5 (the scaling analysis) is the most novel and deserves more depth — specifically, the claim that “familiarity is at ceiling at 1.5B” should be stress-tested with even smaller models (300M, 500M) to find the actual threshold.

Verdict: weak accept — Clean, well-executed study that makes a genuine conceptual contribution (familiarity vs. reliability as distinct scaling phenomena), but limited to one model family in one language, and the “so what for practitioners” story needs sharper development.

Takeaways

1. Stop averaging your hallucination signal. If you’re running semantic entropy or self-consistency to detect hallucinations, you’re paying 5x+ inference cost and still conflating two different failure modes. A single forward pass with a simple IPR/spectral entropy check on MLP activations could flag unfamiliar entities at near-zero marginal cost — then you can spend your compute budget on just the risky queries.

2. Entity familiarity is a free lunch; correctness is not. If you’re building a system that needs to know whether an LLM “knows” an entity (for routing, retrieval augmentation, or abstention), you don’t need a large model. The 1.5B model already has a perfect familiarity signal. Save your GPU budget for the hard part: getting the answer right.

3. The abstention gap is the real research target. The model has the information to refuse, but doesn’t. This suggests that RLHF/fine-tuning for calibrated abstention is where the real leverage is — not in better representation probes or bigger models. If you’re working on hallucination mitigation, focus on the behavioral bottleneck, not the representational one.

4. Post-SwiGLU activations are an underexplored diagnostic surface. Most probing work focuses on residual stream or attention outputs. This paper shows the MLP activation space (specifically after the gating function) carries rich semantic signals accessible with trivial methods. Worth exploring in your own models.