Poisoning the Well: How Public Forums Can Backdoor Language Models

Paper: 2607.15267 Authors: Victoria Graf, Hannaneh Hajishirzi, Noah A. Smith, David Kohlbrenner, Kyle Lo Categories: cs.AI, cs.CL

The Gap

We knew poisoning attacks could work in lab settings—researchers would sneak bad examples into clean datasets like Wikipedia and show models learned harmful behaviors. But that’s like testing a city’s water supply by contaminating a single pristine mountain spring. Real pretraining data isn’t like that; it’s a murky river fed by billions of web pages, messy forums, and social media posts. Prior work ignored two big things: (1) the sheer scale and diversity of actual pretraining corpora, and (2) the data curation pipeline that filters and cleans web crawls before training. This paper asks: can you poison the model not by touching the data directly, but by injecting content into public websites that get crawled? And can we even tell if the poison survives the curation process?

Problem: LMs can be poisoned
    |
Assumption: Prior attacks used small, clean datasets (Wikipedia)
    |
Limitation: Doesn't match real web-scale pretraining
    |
Method: Inject via public forums + measure with HalfLife
    |
Evidence: Poison persists through crawling/curation
    |
Conclusion: Third-party webpages are a viable attack vector

The Increment

One sentence: Before this paper, we thought poisoning required direct access to clean datasets; after, we realize anyone can potentially poison a future LM by posting malicious content in public forums that get web-crawled.

Core Mechanism

The attack has two parts: (1) injecting poison into public discussion interfaces (like forums, comment sections, Q&A sites), and (2) measuring whether that poison actually makes it into the final training data after crawling and curation. For injection, they don’t need special access—just post content where web crawlers will find it. For measurement, they create HalfLife, a tracking system that uses “synthetic canary documents”—carefully crafted posts with unique identifiers that let them trace whether these specific posts survive the data pipeline.

The HalfLife system works by embedding invisible watermarks in forum posts. They post these canaries to real public forums, then track them through the crawling, deduplication, and filtering steps that build modern pretraining datasets. This lets them compute a “half-life” for their poison: how many posts need to be injected so that one survives all the way to training data. It’s like counting how many messages in bottles you need to throw into the ocean so that one washes up on a specific beach.

[Canary Post] --> Web Crawler
        |
        v
[Raw Web Crawl] --> Deduplication
        |
        v
[Filtered Data] --> Quality Filters
        |
        v
[Pretraining Data] --> HalfLife Measures Survival Rate

Think of it like a water treatment plant. The poison is a contaminant you dump upstream (in public forums). The plant (data curation) has multiple filtration stages (deduplication, quality filters, toxicity classifiers). HalfLife measures how much contaminant makes it through each stage. The paper shows that some contaminants—the right kind of poison—have a long half-life and make it through all filters into the drinking water (training data).

Key Concepts

• Computational Propaganda: Not just fake news—it’s any coordinated, inauthentic content designed to manipulate. In this context, it’s forum posts crafted to teach LMs specific harmful behaviors. Imagine a team posting thousands of seemingly legitimate code examples that all have a hidden backdoor pattern; the model learns to generate code with that backdoor.
• Data Curation Pipeline: The industrial-scale cleaning process for web data. Think of it as a series of nets with different-sized holes: first remove exact duplicates, then low-quality pages, then toxic content. The paper’s key insight is that some poisons are shaped to slip through all these nets.
• HalfLife Analysis: A measurement tool, not an attack. It answers: “If I post 100 poison examples on forums, how many will end up in the training data?” It’s like tracking tagged fish to see how many survive predation, migration, and fishing nets to reach a specific spawning ground.

Framework Shift

Before (mainstream approach):       After (this paper):
                                    
Direct Injection                    Indirect Injection
in Clean Data                       via Public Web
      |                                  |
      v                                  v
[Lab Wikipedia] --> [Train]        [Public Forums] --> [Web Crawl]
      |                                  |
      v                                  v
[Simple Poison]                    [Poison] --> [Curation] --> [HalfLife Measurement]
                                        |
                                        v
                                   [Survival Rate]

From attacking data sources directly to attacking the data supply chain, the core shift is recognizing that the web itself is a controllable input to model training.

Expert Assessment

Problem choice: Excellent. This is a genuine, under-explored gap. Everyone worried about data poisoning in controlled settings, but few asked the practical question: “Can you actually do this at web scale through normal channels?” The timing is perfect—LLMs are being trained on ever-larger web crawls, making this a pressing security concern.

Method maturity: Clever insight over brute force. Using public forums is obvious in hindsight but hadn’t been properly studied. HalfLife is elegant—a measurement tool that doesn’t require reverse-engineering the data pipeline. Simpler approaches (like just scraping forums and checking) wouldn’t give you the causal understanding of why poison survives.

Experimental integrity: Mostly solid. They use real forums, real crawls, and measure actual survival rates. The baselines are fair—they compare different injection strategies and document types. One red flag: they don’t quantify the computational cost of their HalfLife analysis, which matters for real-world monitoring.

Writing quality: The abstract and introduction are clear, but the experimental section gets bogged down in technical details. Rewriting Section 5 (the case studies) as a narrative of “here’s how we poisoned a real forum, here’s what happened” would make the paper more compelling and accessible.

Verdict: weak accept — Important problem, novel measurement approach, but the attack scenarios feel somewhat contrived and the paper doesn’t fully address how defenders should respond.

Takeaways

Practitioners can steal three concrete things:

1. HalfLife as a monitoring tool: Use synthetic canaries to audit your data pipeline. If you’re building a training dataset, you can post unique markers and measure their survival to know what percentage of web content actually makes it in.
2. Poison design principles: The paper identifies what makes forum posts likely to survive curation. If you’re a defender, you need filters that catch these patterns; if you’re a red-teamer, you now have a playbook.
3. Supply chain thinking: Security isn’t just about the final dataset—it’s about the entire web-to-training pipeline. This frames data collection as a potential attack surface that needs its own threat models.