Fact-Checking Without Retrieval: Tapping Into What LLMs Already Know

Paper: 2603.05471 Authors: Artem Vazhentsev, Maria Marina, Daniil Moskovskiy, Sergey Pletenev, Mikhail Seleznyov, Mikhail Salnikov, Elena Tutubalina, Vasily Konovalov, Irina Nikishina, Alexander Panchenko Categories: cs.CL, cs.AI

The Gap

Current fact-checking systems follow a two-stage pipeline: retrieve external evidence from Wikipedia or the web, then use an LLM to verify if the claim matches the evidence. This approach inherits all the failure modes of retrieval — missing documents, outdated indexes, query-document mismatch — while ignoring that the LLM already encoded vast factual knowledge during pretraining. RAG-based methods like FactScore and FActScore treat the model as a dumb verifier that can’t reason without external crutches, even though these same models can answer trivia questions and explain complex concepts from memory.

The boundary being pushed: can we fact-check claims using only what’s already inside the model’s parameters, without touching external databases?

Problem: Retrieval bottleneck in fact-checking
   |
   v
Assumption: LLMs encode verifiable knowledge in internal representations
   |
   v
Method: Probe hidden states instead of output logits
   |
   v
Evidence: 9 datasets, representation-based > logit-based
   |
   v
Conclusion: Retrieval-free fact-checking is viable and generalizes better

The Increment

One sentence: Before this paper, fact-checking meant “retrieve then verify”; after, we can verify directly from model internals, faster and more robust to distribution shift.

Core Mechanism

INTRA (INTernal Representation Analysis) works by treating the LLM as a black box with observable internal states. When you feed a claim like “Paris is the capital of Germany” into the model, each transformer layer produces a hidden representation — a vector encoding what the model “thinks” at that processing stage. Instead of waiting for the final output token probabilities (which are optimized for generation, not verification), INTRA trains lightweight classifiers on these intermediate representations.

The key insight: different layers encode different types of knowledge. Early layers capture surface patterns, middle layers encode semantic relationships, and late layers prepare for generation. By extracting representations from multiple layers and training a probe to classify “true” vs “false” based on these vectors, INTRA learns to read the model’s internal “confidence” about factual claims. The method also introduces interaction terms — it doesn’t just look at layer 12 and layer 24 separately, but computes how they relate (via element-wise products), capturing whether different reasoning stages agree or conflict.

The architecture has three components: (1) a frozen LLM that processes claims, (2) representation extractors that pull hidden states from selected layers, and (3) a trainable classifier head (logistic regression or shallow MLP) that maps these representations to verification decisions. Crucially, the LLM weights never update — you’re just learning to interpret what’s already there.

Claim: "Paris is capital of Germany"
   |
   v
+---------------------------+
|   Frozen LLM (e.g. Llama) |
|                           |
|  Layer 8:  [h_8 vector]   |---+
|  Layer 16: [h_16 vector]  |---+---> Representation
|  Layer 24: [h_24 vector]  |---+     Extractor
+---------------------------+       |
                                    v
                            +----------------+
                            | Interactions:  |
                            | h_8 * h_16     |
                            | h_16 * h_24    |
                            | Concatenate    |
                            +----------------+
                                    |
                                    v
                            +----------------+
                            | Classifier     |
                            | (LogReg/MLP)   |
                            +----------------+
                                    |
                                    v
                              [FALSE: 0.92]

Think of INTRA like reading someone’s facial expressions instead of waiting for them to speak. When you ask a friend “Is Paris in Germany?”, you don’t need to hear their answer — you can see the confusion or certainty in their face immediately. The LLM’s hidden layers are like micro-expressions at different moments: the initial reaction (early layers), the processing of implications (middle layers), and the preparation to respond (late layers). INTRA trains a “facial expression reader” that looks at these micro-expressions across multiple moments and learns patterns like “when layer 12 shows high activation in dimension 847 AND layer 20 shows low activation in dimension 1203, the model knows this claim is false.” The interaction terms capture whether the expressions are consistent — if early layers look confident but late layers look uncertain, that’s a red flag.

Key Concepts

• Parametric knowledge vs retrieval-augmented knowledge: Imagine you’re taking a test. Parametric knowledge is what you memorized and can recall from your brain. Retrieval-augmented knowledge is when you’re allowed to use your textbook during the test. Current fact-checking systems always open the textbook (retrieve from Wikipedia), even for questions you already know. This paper asks: what if we just test what you memorized first? The LLM’s parameters (billions of numbers learned during training) encode facts like “Paris is France’s capital” implicitly. You don’t need to retrieve a Wikipedia article about Paris if the model already “knows” this from seeing it thousands of times during pretraining. The trade-off: parametric knowledge can be outdated (training data has a cutoff date) and might hallucinate, but it’s instant and doesn’t fail when retrieval breaks.

• Hidden representations vs output logits: When an LLM generates text, the final step is converting a hidden vector into probabilities over vocabulary tokens (logits). Most fact-checking methods look at these logits — does the model assign high probability to “true” or “false”? But logits are optimized for fluent generation, not truthfulness. Hidden representations are the intermediate vectors inside the model before this final conversion. Think of it like this: logits are what someone says out loud, but hidden representations are their internal thoughts before speaking. A person might confidently say something false (high logit for wrong answer) while their internal reasoning is confused (messy hidden representation). By probing hidden states, you access the “thinking process” rather than just the polished output. This paper shows that hidden representations are more reliable signals for fact verification.

• Interaction between layers: Neural networks process information hierarchically — early layers detect simple patterns, later layers combine them into complex concepts. In fact-checking, layer 5 might encode “Paris is a city”, layer 15 might encode “Germany is a country”, and layer 25 might encode “capitals are in their respective countries”. The interaction term (element-wise multiplication of layer 5 and layer 15 representations) captures whether these two pieces of knowledge are being activated together consistently. If the model activates “Paris” neurons and “Germany” neurons simultaneously but NOT the “capital-of” relationship neurons, the interaction term will show this mismatch. It’s like checking if different parts of your brain agree — if your visual cortex sees a cat but your language center is preparing to say “dog”, the interaction reveals the inconsistency.

Framework Shift

Before (mainstream approach):        After (this paper):

Claim                                Claim
  |                                    |
  v                                    v
+-------------+                      +-------------+
| Retrieval   |                      |   LLM       |
| System      |                      | (frozen)    |
+-------------+                      +-------------+
  |                                    |
  v                                    v
Evidence docs                        Hidden states
  |                                  (h_8, h_16, h_24...)
  v                                    |
+-------------+                        v
| LLM reads   |                      +-------------+
| evidence    |                      | Probe       |
| and judges  |                      | classifier  |
+-------------+                      +-------------+
  |                                    |
  v                                    v
Verdict                              Verdict

External dependency                  Self-contained
Fails if retrieval fails             Always available
Slow (retrieve + verify)             Fast (one forward pass)

From “trust but verify externally” to “trust the model’s internal knowledge”, the core shift is treating the LLM as a knowledge base itself rather than just a reasoning engine that needs external facts.

Expert Assessment

Problem choice: This is a real gap, not manufactured. The field has been so focused on RAG that we’ve ignored the elephant in the room — these models already know a lot. The timing is right: as LLMs get larger and more capable, their parametric knowledge becomes more reliable, making retrieval-free approaches viable. It sits at an interesting inflection point where model scale meets practical deployment constraints (retrieval is expensive and fragile).

Method maturity: INTRA is more clever insight than brute force. The interaction terms are elegant — they add minimal parameters but capture non-linear relationships between layers. However, I’m skeptical about one thing: the paper doesn’t deeply explore which layers matter most or why. The method feels like “try all layer combinations and see what works”, which is fine for a first paper but leaves theoretical understanding thin. A simpler approach might be: just use the last layer representation with a better probe architecture. The authors don’t convincingly rule this out.

Experimental integrity: The baselines are fair and comprehensive — 18 methods is thorough. The generalization tests (long-tail knowledge, multilingual, different claim sources) are exactly what you’d want to see. One red flag: the paper doesn’t report computational costs clearly. How much does training the probe cost? How does inference speed compare to retrieval-based methods? These numbers matter for practical adoption. Also, the datasets are mostly short factoid claims. What happens with complex, multi-hop reasoning claims? The paper hints at this with “long-form generation” but doesn’t stress-test it enough.

Writing quality: The abstract and intro are strong, but the method section is dense and could use more intuition-building before diving into formulas. The related work section is thorough but reads like a literature dump — it would benefit from a clear taxonomy of approaches rather than a chronological list. If I were rewriting one section, it’d be the results: the tables are overwhelming with too many numbers. A clearer narrative about which findings matter most would elevate the whole paper.

Verdict: Weak accept — solid contribution that opens a new research direction, but the method feels somewhat incremental (probing isn’t new, the innovation is applying it systematically to fact-checking) and the evaluation could be more rigorous on computational costs and complex reasoning.

Takeaways

For practitioners, here’s what’s immediately useful:

1. Probe hidden states, not just logits: If you’re building any kind of truthfulness detector or quality filter for LLM outputs, don’t just look at output probabilities. Extract hidden representations from multiple layers and train a lightweight classifier. This transfers beyond fact-checking — it works for detecting hallucinations, measuring uncertainty, or filtering low-quality generations.

2. Interaction terms are cheap and effective: Element-wise multiplication of representations from different layers adds almost no parameters but captures non-linear relationships. This trick applies anywhere you have hierarchical features — not just LLMs, but any deep network where you want to check if different processing stages agree.

3. Retrieval-free as a first-pass filter: You don’t have to choose between retrieval and parametric knowledge. Use INTRA-style methods as a fast first pass: if the model’s internal representations show high confidence, skip retrieval. Only retrieve for uncertain cases. This hybrid approach could dramatically reduce retrieval costs in production systems.

4. Generalization matters more than in-distribution performance: The paper’s emphasis on testing across different claim sources, languages, and knowledge distributions is a template for how to evaluate any ML system meant for real-world deployment. Don’t just report accuracy on your test set — show robustness to distribution shift.

The core transferable idea: when you have a pretrained model, its internal states often contain richer signals than its outputs. Learn to read those signals.