Reducing Belief in Conspiracy Theories As They Unfold

Paper: 2608.06151 Authors: Thomas H. Costello, Nathaniel Rabb, Michael Nicholas Stagnaro, Gordon Pennycook, David Rand Categories: cs.HC, cs.AI

The Gap

There is now solid evidence that conversational dialogue with an LLM can reduce belief in conspiracy theories. That finding was surprising when it landed, because it cut against the dominant account of conspiracy belief as identity-protective and therefore immune to evidence. It turned out that tailored, patient, factually specific counter-argument moves people — and that an LLM can supply it at scale in a way a human interlocutor cannot.

But every demonstration of this ran on established conspiracy theories: the moon landing, 9/11, vaccines. That setting has a property doing quiet work in the result. An established conspiracy theory has an established rebuttal. Decades of journalism, investigation, and debunking sit in the training data, so the model is not reasoning its way to a counter-argument — it is retrieving a well-developed one and tailoring the delivery.

This leaves the case that actually matters untested. Conspiracy theories are most consequential and most contagious in the first days after a major public event, before facts are settled and while the interpretive frame is still being fixed. That window has none of the properties the earlier work relied on. There is no debunking literature. There may not yet be an authoritative account of what happened. The model’s own training data may predate the event entirely. And beliefs formed in that window are the ones that harden and propagate.

So the open question is not “can LLMs debunk conspiracies” — that is answered — but whether the effect survives the removal of the scaffolding that made it easy.

Established finding: LLM dialogue reduces belief
in well-known conspiracy theories
         |
         v
Unexamined enabling condition:
those theories have decades of debunking in training data
         |
         v
[This paper] Remove the scaffolding.
Test conspiracies forming in the days after a crisis
         |
         +-------------------------+
         |                         |
         v                         v
   July 2024                  Sept 2025
   Trump assassination        Kirk assassination
   attempt (N = 472)          (N = 1035)
         |                         |
         +-----------+-------------+
                     |
                     v
    LLM dialogue  vs  irrelevant-topic LLM chat
                  vs  static fact sheet
                     |
                     v
    Result: significant belief reduction in BOTH
                     |
                     v
    Follow-up 1-2 months later: reduced belief in
    DIFFERENT conspiracies after LATER crisis events

The Increment

One sentence: Before this paper, we knew LLMs could talk people out of conspiracy theories that already had a debunking record; after it, we know the effect works on conspiracies that are still forming, and that it generalizes to other conspiracies months later.

Core Mechanism

The design is a field experiment run against the clock, and the timing is the methodological contribution. Both studies launched in the days immediately following a real crisis event — the July 2024 assassination attempt on Donald Trump, and the September 2025 assassination of Charlie Kirk. U.S. adults who held conspiratorial views about the event were recruited and assigned to a multi-turn conversation with an LLM prompted to reduce their conspiracy belief.

The control conditions are chosen to isolate two different alternative explanations, and this is where the design earns its keep. One control has participants discuss an irrelevant topic with an LLM. This absorbs everything about the experience that is not the argument: the novelty of talking to an AI, the attention and engagement, the experience of being listened to, any general reflective effect of a long conversation. The other control shows a static fact sheet — the same corrective information, delivered without dialogue. This isolates the interactive component specifically: if a fact sheet worked as well, the whole LLM apparatus would be an expensive way to deliver text. The treatment beat both, in both experiments.

The most interesting result is the one that was not the primary hypothesis. Participants were followed up one to two months later, in the wake of subsequent crisis events, and showed reduced belief in different conspiracy theories. This is a different kind of claim from the main effect. Belief in the target conspiracy going down is consistent with a narrow reading — the model supplied specific counter-evidence about a specific event, and it landed. Belief in unrelated later conspiracies going down is not explicable that way. Something transferred that was not event-specific content: some disposition, some raised threshold, some habit of interrogating a claim before adopting it.

   Timeline of Experiment (both replications)

   crisis event occurs
         |
         |  days
         v
   recruit adults holding conspiratorial views
         |
    +----+--------------------+--------------------+
    |                         |                    |
    v                         v                    v
 TREATMENT              CONTROL A            CONTROL B
 multi-turn LLM         LLM chat about       static
 dialogue, prompted     irrelevant topic     fact sheet
 to reduce belief            |                    |
    |                        |                    |
    |   isolates:            |                    |
    |   the ARGUMENT         |                    |
    |                   isolates:            isolates:
    |                   AI novelty +         information
    |                   engagement           without dialogue
    |                        |                    |
    +----+--------------------+--------------------+
         |
         v
   immediate measure: belief in target conspiracy
   --> treatment significantly lower (both experiments)
         |
         |  1-2 months, NEW crisis events occur
         v
   follow-up measure: belief in DIFFERENT conspiracies
   --> still reduced

Here is the structural metaphor. Think of conspiracy belief as an immune response to an ambiguous event. When something shocking and unexplained happens, the mind needs an account fast, and in the absence of a settled one it will manufacture something that fits the available fragments and the person’s priors. The account, once adopted, resists revision — that is the identity-protective story the field long believed.

Earlier LLM-debunking work was, in this metaphor, treating a chronic infection with a well-characterized drug. The conspiracy was old, the counter-arguments were thoroughly developed, and the model’s job was to administer a known therapy with unusual patience and personalization.

This paper does something different: it intervenes during the acute phase, before the belief has consolidated, using no pre-existing therapy because none exists yet. And the surprising finding is not just that intervening early works — it is that the effect looks less like a drug and more like a vaccine. Months later, facing a new pathogen — a different crisis, a different conspiracy — the treated participants respond less strongly. Whatever the conversation did, it did not merely neutralize one belief; it left something behind that engaged with the next one. That is why the downstream result is the paper’s most important sentence, and also its most fragile one.

Key Concepts

• Unfolding versus established conspiracies: The distinction the whole paper rests on. An established conspiracy has three things an unfolding one lacks: a documented factual record, a body of existing rebuttals in the model’s training data, and a belief that has already hardened in the holder. Removing all three at once is a much harder test, and it is also the test that corresponds to real deployment — nobody needs an intervention against moon-landing denial with the urgency they need one in the 72 hours after a national shock. A concrete illustration of the difficulty: for the September 2025 event, a model may have no training data about the event whatsoever, so it cannot supply facts about what happened. Whatever it is doing must be more procedural than factual.

• The two-control design: Why a single control would have been uninformative. Comparing LLM dialogue against nothing would confound the argument with the experience. People who spend twenty minutes in an engaged conversation, with a novel and attentive interlocutor, about a topic they care about, may moderate their views for reasons having nothing to do with what was argued — social desirability, reflection, or simple fatigue with the position. The irrelevant-topic control absorbs that. Separately, comparing against nothing would leave open that the model’s content was all that mattered, in which case a PDF would do. The fact-sheet control absorbs that. Beating both is what licenses the claim that interactive, tailored counter-argument is the active ingredient.

• Downstream transfer: Reduced belief in different conspiracies, following different crisis events, one to two months later. The reason this is the most theoretically loaded result: it is inconsistent with the intervention working purely through content delivery. If the mechanism were “the model told me true things about event X,” there is no route by which that lowers belief about unrelated event Y months later. The result therefore points toward something dispositional — a shift in how readily a person adopts a conspiratorial frame at all. It is also the result most exposed to alternative explanations, since repeat participation in a study about conspiracy beliefs is itself an intervention.

Framework Shift

Before (prior LLM-debunking work):    After (this paper):

  established conspiracy               conspiracy forming NOW
  (decades old)                        (days old)
        |                                    |
        v                                    v
  rich debunking record                no debunking record;
  in training data                     event may postdate
        |                              training data
        v                                    |
  model retrieves + tailors                  v
  a known counter-argument             model must work
        |                              procedurally, not
        v                              from stored rebuttals
  belief in THAT theory drops                |
        |                                    v
        v                              belief drops -- AND
  (transfer untested)                  transfers to OTHER
                                       conspiracies months later
                                             |
                                             v
                                       looks less like a cure,
                                       more like a vaccine

From showing that LLMs can deliver mature counter-arguments to old conspiracy theories, to showing they can move beliefs in the hours when beliefs are actually being formed, the core shift is from debunking as retrieval to debunking as a real-time, and apparently durable, cognitive intervention.

Expert Assessment

Problem choice: Excellent, and the field-timing is the reason. The gap between “works on the moon landing” and “works on the thing everyone is arguing about right now” is exactly the gap that determines whether the earlier finding has any practical consequence, and it is a gap most researchers would leave open because closing it requires being ready to launch a study within days of an unpredictable event. Running it twice, on two separate real crises fourteen months apart, is not just replication — it is evidence that the team built standing capacity to do this, which is itself a contribution to how this research area can be done. The paper sits at the point where the LLM-persuasion literature stops being an interesting curiosity and starts being a policy question.

Method maturity: The two-control design is the right design and is executed with unusual discipline. Choosing an irrelevant-topic LLM conversation as one arm shows the authors anticipated the strongest deflationary reading of their own result, which is the mark of a serious experimental group. What the design cannot do — and this is inherent rather than a criticism of execution — is tell us which feature of the dialogue matters. The treatment is a bundle: personalization, patience, interactivity, specific evidence, non-judgmental tone. Beating a fact sheet establishes that dialogue beats text; it does not establish which property of dialogue is load-bearing, and that is what anyone trying to deploy or defend against this would most need to know.

Experimental integrity: Sample sizes are respectable and the near-tripling from Experiment 1 to Experiment 2 suggests the second was appropriately powered on the basis of the first. Self-reported belief measures remain the field’s standard and the field’s weakness: demand characteristics are a live concern when a participant has just spent twenty minutes with an interlocutor who was visibly trying to change their mind, and the natural response is to report movement. The irrelevant-topic control does not fully absorb this, since only the treatment arm makes the experimenter’s goal legible. The downstream result is where I would apply the most pressure. It is the paper’s most striking claim and it rests on a follow-up sample that has necessarily attrited, and where the participants have been primed by the original study to think about conspiracy beliefs. Attrition in this setting is unlikely to be random — people who found the conversation persuasive may be likelier to return. None of this makes the result wrong, but it is the claim that most needs a preregistered, adversarially designed replication before it is treated as established.

Writing quality: The abstract is clear and admirably restrained given how quotable the finding is; “shed light on the psychology of emerging conspiracies” is a more modest framing than the result would license, which is the right instinct. The corner cut is that the downstream transfer result is reported in a single sentence and carries a disproportionate share of the paper’s theoretical weight. A section that took the alternative explanations for that result seriously — attrition, priming, regression to the mean — would strengthen the paper more than any additional experiment.

Verdict: strong accept — The hard version of an important question, answered twice on real events with controls that anticipate the obvious objections; the durable-transfer claim needs more scrutiny than it gets, but the main result is solid and consequential.

Takeaways

Three things a practitioner can steal:

1. Two controls, one for the experience and one for the content. This design pattern transfers directly to evaluating any LLM intervention — tutoring, therapy-adjacent tools, customer support, behavior change. Run one arm where the model does something irrelevant (absorbing novelty, attention, and engagement effects) and one arm delivering the same information statically (absorbing the content). Only what beats both is attributable to interactive, tailored generation. Most product-side evaluations of LLM features run neither control and therefore cannot distinguish their model from a well-written FAQ.

2. Test the capability where the scaffolding is absent. The generalizable methodological move: identify the condition that made your earlier result easy — abundant training data, a settled ground truth, a well-documented answer — and construct the case where it is missing. This is the difference between a capability that benchmarks well and one that survives deployment, and it applies to retrieval systems, agents, and evaluation suites as much as to persuasion. Ask what your system is retrieving rather than reasoning, then remove the thing it retrieves.

3. Distinguish content effects from dispositional effects by looking for transfer. If an intervention only changes beliefs about the thing it argued about, it is a content effect and must be re-delivered for every new instance. If it changes beliefs about unrelated things later, something dispositional moved, and the economics are entirely different — one interaction with lasting spillover rather than perpetual per-case delivery. Measuring off-target outcomes at follow-up is cheap and is the only way to tell these apart.