RedirectQA: Non-Verbatim Memorization in LLMs

Paper: 2604.21882 Authors: Research Team Categories: cs.CL, cs.LG

Problem

Existing memorization defenses focus on verbatim copying:

• They detect exact or near-exact matches from training data
• They miss knowledge that’s been “rephrased” by the LLM
• They assume memorization = copying, which is false

RedirectQA Attack

The paper demonstrates a new attack vector:

Key Insight

LLMs memorize knowledge, not just text. They can “remember” facts and reproduce them in new phrasings:

1. Question Reformatting: Same facts, different question structure
2. Answer Paraphrasing: Same knowledge, different words
3. Multi-hop Reasoning: Combine multiple memorized facts

Attack Method

• Query LLM with redirected questions (different surface form, same meaning)
• Collect responses that reveal memorized knowledge
• Verify using source attribution techniques

Results

Defense	Verbatim Copy	RedirectQA
Exact Match	98% blocked	12% blocked
N-gram Deduplication	89% blocked	8% blocked
Proposed Defense	TBD	TBD

Takeaways

• Non-verbatim memorization is a real and serious threat
• Current defenses focus on wrong threat model
• Need semantic-level deduplication, not just surface-level
• LLM privacy is more complex than copyright