Diffusion LLMs as Both Victim and Weapon: Safety Neurons Travel Across Architectures

Paper: 2608.07430 Authors: Elena Dumitrescu, Gert Lek, Lydia Y. Chen, Jérémie Decouchant Categories: cs.LG, cs.AI

The Gap

Two separate literatures had grown up next to each other without touching.

On one side, mechanistic safety work on autoregressive (AR) models had converged on an uncomfortable finding: alignment is sparse. A small number of MLP neurons or a single “refusal direction” in residual space carries most of the refusal behavior, and if you prune or ablate them, the model happily complies. Wei et al.’s pruning work and the refusal-direction line of work (Arditi et al.) both land here. But every one of these studies assumes left-to-right generation.

On the other side, diffusion LLMs (LLaDA, Dream, Fast-dLLM) had arrived as a genuinely different generation paradigm — iterative parallel denoising over a masked sequence instead of next-token prediction. Safety work on them was almost entirely behavioral: throw known jailbreak prompts at them, measure attack success rate (ASR), observe that they are somewhat more fragile. Nobody had asked the mechanistic question: where does the refusal live inside a denoiser, and is it the same place it lives in an AR model?

That question has real bite, because of a supply-chain fact: Dream-7B was not trained from scratch — it was initialized from Qwen2.5-7B. If safety is sparse and localized, and the diffusion model inherited its weights, then the coordinates of the safety circuit might survive the conversion from AR to diffusion. That would make a safety audit of Qwen a free attack map for Dream.

The second, less obvious gap: existing jailbreak generators are expensive. GCG needs white-box gradients and thousands of forward passes; PAIR/TAP/AutoDAN need an attacker LLM in the loop and hundreds of queries against the live target. Nobody had noticed that a diffusion LLM is structurally a constrained-text optimizer — it revises a whole sequence in place, in any order — which makes it an unusually good machine for searching prompt space offline.

   PRIOR STATE
   +----------------------------+   +---------------------------+
   | AR mech-interp: safety is  |   | DLLM safety: behavioral   |
   | sparse, prunable, local    |   | only, no circuit-level    |
   +-------------+--------------+   +-------------+-------------+
                 |                                |
                 +---------------+----------------+
                                 v
   PROBLEM . . can we locate safety circuitry inside a denoiser?
                                 |
                                 v
   ASSUMPTION  weight inheritance (Qwen2.5 -> Dream) preserves
               the *coordinates* of the safety footprint
                                 |
              +------------------+-------------------+
              v                                      v
   METHOD (A) safety-neuron ID +          METHOD (B) SN-Guided
   self-prune / transfer-prune            Diffusion: denoise the
   across architectures                   prompt to minimize the
              |                           safety-neuron loss
              v                                      |
   EVIDENCE   LLaDA 2.6 -> 73.8 ASR                  v
              Dream  1.9 -> 86.6 (self)     EVIDENCE AUROC = 1.0
              Dream  1.9 -> 73.2 (from Qwen)         77.1% Llama-3-8B
              Fast-dLLM 7.0 -> 86.3                  86.9% Qwen2.5-7B
              |                                      74.3% Gemini-Flash-Lite
              |                                      20 episodes / prompt
              +------------------+--------------------+
                                 v
   CONCLUSION  DLLM alignment is a thin, *portable* layer; and the
               same map that breaks it turns the DLLM into a cheap
               black-box attack engine against AR and closed models

The Increment

One sentence: Before, diffusion LLM safety was a black box you probed by prompting; after, it is a *located, sparse, and architecture-portable set of neurons — and the diffusion process itself becomes the search engine that routes around them.

Core Mechanism

There are really two artifacts in this paper, and they chain together. The first is a safety-neuron (SN) locator plus pruner. You run a set of harmful prompts and a matched set of benign prompts through the model, record MLP activations, and score each neuron by how much more it fires on the harmful set (an activation-contrast / attribution score). Take the top-k, zero them out, and re-measure ASR. Doing this on a DLLM’s own neurons is “self-pruning.” The interesting move is transfer pruning: locate the neurons in the *AR ancestor (Qwen2.5-7B), map the indices directly onto the diffusion descendant (Dream, and Fast-dLLM built on that lineage), and prune those. If ASR jumps, the safety footprint survived the AR-to-diffusion conversion — which is the mechanistic claim.

The second artifact, SN-Guided Diffusion, inverts the direction. Instead of editing weights (which requires access you usually do not have), it uses the SN scores as an *objective for text search. The safety-neuron loss is a weighted sum of activations over the identified safety neurons for a candidate prompt: high value means “this text is tripping the refusal circuit,” low value means “this text reads as benign to the safety machinery while still carrying the harmful request.” Now recall what a masked diffusion model does: it starts from a fully masked sequence and, over T denoising steps, unmasks tokens in arbitrary order, with every step able to revise its plan for the whole sequence. So at each step you score candidate unmaskings by the SN loss and keep the low-loss ones. After ~20 generation episodes you have a prompt that sits in a low-safety-activation region of a surrogate model’s representation space. Then you fire it at the real target — Llama-3, Qwen2.5, Gemini-2.5-Flash-Lite — with zero prior queries. That is what “fully offline black-box” means: the optimization loop never touches the victim.

The AUROC = 1.0 number is the internal sanity check: the SN loss alone perfectly separates benign prompts from the generated jailbreaks, i.e. the signal being optimized is a real, sharp axis rather than noise.

  PHASE 1: LOCATE
  harmful set --+                              +-> per-neuron
                +--> [ surrogate model ] ------+   contrast score
  benign set ---+      MLP activations             |
                                                   v
                                          top-k SAFETY NEURON map
                                          (indices + weights)
                                              |            |
                    +-------------------------+            +---------+
                    v                                                v
  PHASE 2a: PRUNE (white-box)              PHASE 2b: SN-GUIDED DIFFUSION
  zero the mapped neurons                  (black-box, offline)
   self  : LLaDA -> LLaDA                   masked prompt: [M][M][M]...[M]
   xfer  : Qwen2.5 -> Dream                              |
                    |                        +-----------+------------+
                    v                        |  step t: propose        |
             ASR 2.6% -> 73.8%               |  unmaskings in parallel |
                                             |           |             |
                                             |           v             |
                                             |  score with weighted    |
                                             |  SN loss (low = quiet)  |
                                             |           |             |
                                             |     keep low-loss       |
                                             +-----------+------------+
                                                         | x20 episodes
                                                         v
                                              final prompt (never
                                              tested on target yet)
                                                         |
                                                         v
                                        +----------------+---------------+
                                        v                v               v
                                   Llama-3-8B      Qwen2.5-7B     Gemini-Flash-Lite
                                     77.1%            86.9%           74.3%

The metaphor: think of alignment as a burglar alarm with surprisingly few motion sensors.

Phase 1 is stealing the floorplan: by watching which sensors chirp when someone walks the “harmful” corridor versus the “benign” corridor, you learn that only a dozen sensors out of thousands actually matter. Transfer pruning is the punchline about the builder: Dream is not a new building, it is Qwen2.5 gut-renovated. The walls moved, the way people walk through it changed completely (parallel instead of single-file), but the sensors were never re-installed — they are still bolted to the same coordinates. So a floorplan stolen from Qwen opens Dream.

Phase 2b is where the burglar stops cutting wires and starts route-planning. Cutting wires (pruning) needs to be inside the building already, which is the whole weakness of white-box attacks. Instead, the burglar takes the sensor map home and works on a sketchpad with an eraser: this is the diffusion process. An autoregressive attacker has to plan the route one footstep at a time and can never go back and change footstep three; a diffusion attacker sketches the entire route at once and keeps erasing and redrawing any part of it. Each draft gets scored against the stolen sensor map — “how loud is this route?” — and after twenty drafts you have a path that the sensors, as far as the map predicts, will sleep through. Only then do you walk into the real building. And because the same architecture firm wired half the buildings on the street, the quiet route generalizes to Llama-3 and to Gemini, whose floorplan you never had.

The load-bearing part of the metaphor is the eraser. Everything expensive about GCG and PAIR comes from not having one: they must either brute-force gradients or keep knocking on the victim’s door to see if it opens. Diffusion gives you free, arbitrary-order revision, so the search becomes cheap and can happen entirely off-premises.

Key Concepts

• Safety neuron: Inside a transformer’s feed-forward layers there are hundreds of thousands of individual units, each of which outputs a single number per token. Most encode boring things — “this is a plural noun,” “we’re in a Python block.” A handful behave like a smoke detector: they light up specifically when the input is a request the model was trained to refuse, and their output feeds the pathway that produces “I can’t help with that.” You find them by simple bookkeeping: run 200 harmful prompts and 200 harmless ones, and rank neurons by how lopsided their average activation is. The unsettling empirical fact — established before this paper, for AR models — is that zeroing out on the order of dozens to hundreds of these out of ~10^5 flips a well-aligned model into a compliant one. Alignment turns out to be a thin veneer bolted onto a capable model, not a property diffused through it.

• Masked diffusion decoding (and why it is an optimizer): An AR model writes like someone typing with no backspace key: token 1, then token 2 conditioned on token 1, forever forward. A masked diffusion LLM starts with a row of blanks — [MASK] [MASK] [MASK] [MASK] — and on each pass fills in whichever blanks it is most confident about, re-reading the whole partially-filled sequence each time. Think of solving a crossword: you don’t fill 1-Across then 2-Across in order, you fill what you’re sure of and let it constrain the rest, and you erase when a guess stops fitting. That “erase and refill anywhere” property is exactly what a discrete optimizer needs, which is why the paper can treat generation as a *search under a loss rather than just sampling.

• Offline black-box transfer: Three separate constraints that jailbreak papers usually trade off. *Black-box: no gradients from the victim. Offline: no queries to the victim during optimization — you cannot even use pass/fail feedback. Transfer: the prompt was tuned against a stand-in and must work on a model it has never seen. GCG is white-box; PAIR is black-box but very much online (hundreds of victim queries per prompt); this paper claims all three at once, with 20 episodes of surrogate generation as the entire budget. The reason that is even conceivable is the phase-1 finding: if the safety circuit is shared across the model family, then a surrogate’s refusal geometry is a decent proxy for the victim’s.

Framework Shift

  Before (mainstream approach):          After (this paper):

  DLLM safety studied behaviorally       DLLM safety studied mechanistically
  +---------------------+                +-------------------------+
  | known AR jailbreaks |                | activation contrast on  |
  |         |           |                | harmful vs benign       |
  |         v           |                |         |               |
  |    [ DLLM  ] --> ASR|                |         v               |
  +---------------------+                | SN map = shared asset   |
   "is it more fragile?"                 |    /            \       |
                                         |   v              v      |
  Jailbreak generation:                  | prune          guide    |
  +---------------------+                | (audit)      (attack)   |
  | attacker LLM /      |                +-------------------------+
  | gradient search     |                 "where is it, and does
  |    |     ^          |                  it move between models?"
  |    v     | pass?    |
  | [ VICTIM ]----------|                Generation as search:
  | 100s-1000s queries  |                +--------------------------+
  +---------------------+                | [M][M][M] --+            |
   attack cost tied to                   |     ^       | SN loss    |
   victim access                          |     +-------+ (offline)  |
                                         |         |                |
  DLLM = a model to defend               |         v x20            |
                                         | prompt --> ANY victim    |
                                         +--------------------------+
                                          DLLM = the attack engine

One sentence: From “diffusion LLMs are a new class of model we should test for safety” to “diffusion LLMs are a new class of *optimizer whose inherited safety circuitry is both the lock and the lockpick” — the core shift is treating the safety footprint as a portable coordinate system rather than a per-model property.

Expert Assessment

Problem choice: A real gap, and well-timed. The weight-inheritance angle is the genuinely sharp observation — the field has been publishing AR interpretability results for two years while a new model family quietly bootstrapped off those same checkpoints, and nobody asked whether the audit results transfer. That is the kind of question that looks obvious only after someone asks it. The dual framing (“targets and adversaries”) is more editorial than scientific, but it does hold: both halves depend on the same SN map, so it is not two papers stapled together. Where it sits in the trajectory: a natural and probably necessary extension of the sparse-alignment line into a new architecture, not a paradigm-level contribution.

Method maturity: Mixed. Phase 1 is inherited machinery — activation-contrast neuron scoring and pruning-based alignment breaking are both established; the increment is the index-mapping trick, which is conceptually two lines of code and worth exactly as much as the empirical result it produces (which is a lot, but let’s be clear about the intellectual cost). Phase 2 is the clever part: recognizing that any-order denoising is a discrete search procedure, and that an interpretability artifact makes a serviceable reward. The “20 episodes” efficiency claim is the strongest selling point. What is being overlooked: a straightforward baseline is to skip the neuron machinery entirely and use the surrogate’s refusal direction (a single vector, cheaper to extract, well-validated) as the loss. If refusal-direction guidance gets within a few points of SN guidance, the neuron-level apparatus is decoration. I would want that ablation front and center.

Experimental integrity: Several things to squint at. First, AUROC = 1.0 should always raise an eyebrow, not a cheer. Perfect separability on a designed benchmark usually means the two classes differ in some surface statistic, and here it cuts against the paper’s own interests: if the SN loss perfectly separates benign prompts from these jailbreaks, then the SN loss is a *perfect detector, and the attack ships with its own defense. The paper needs to either report that honestly as a limitation or show that an adaptive version defeats SN-based screening. Second, pruning results need utility controls. Zeroing neurons raises ASR partly by making the model less discriminating overall; without MMLU/GSM8K/coherence numbers and a random-neuron control matched for capability drop, “2.6% -> 73.8%” is not yet evidence of a *safety-specific circuit. Third, ASR numbers depend entirely on the judge; refusal-substring matching inflates badly, and LLM-judge protocols vary. Fourth, the threat models are not comparable and should not be reported side by side without a caption warning: self-pruning requires full weight access (a strong assumption, and arguably a defender’s audit rather than an attack), while SN-Guided Diffusion is genuinely black-box. Fifth, 74.3% on Gemini-2.5-Flash-Lite is a claim with a shelf life measured in weeks; without the exact model snapshot and dates, it is not reproducible. Finally, the cost comparison against GCG/PAIR needs matched budgets and a matched judge, or “orders of magnitude cheaper” is comparing a bicycle to a boat.

Writing quality: The abstract is doing the classic thing of substituting a wall of numbers for a claim. The mechanistic argument — the actual contribution — is where the corner-cutting will show: showing that ASR rises after transfer pruning is *consistent with a shared safety footprint but does not establish it. What would elevate the whole paper is a rewritten Phase 1 that reports the concrete overlap statistics (Jaccard of top-k safety neurons between Qwen2.5 and Dream, versus a matched-baseline pair with no shared ancestry, plus LLaDA as the trained-from-scratch control), and layer-wise distributions. That single section, done properly, turns an attack paper into a finding about how safety survives architectural conversion — which is the more durable result. Also, a limitations section that owns the detector duality would make the paper harder to attack, not easier.

Verdict: weak accept — the weight-inheritance transfer finding and the reframing of denoising as a cheap offline attack search are both genuinely new and useful, but the mechanistic claim is currently supported by downstream ASR rather than direct circuit evidence, and the AUROC = 1.0 result quietly undermines the threat model it is meant to support.

Takeaways

• Check your model’s ancestry before trusting its safety evaluation. If a model was initialized from another model’s weights, treat published interpretability findings on the ancestor as applying to the descendant until proven otherwise. This generalizes well past diffusion: distilled models, continued-pretraining forks, MoE-ified conversions, and quantized redistributions all likely inherit sparse circuit locations. It is a supply-chain security argument, and it applies to *capability circuits too, not just safety.
• Interpretability artifacts are reward functions. The reusable pattern is: extract a sparse, interpretable signal from a surrogate, then use it as an offline loss for search. This works for anything you can localize — hallucination-associated features, sycophancy, tool-call triggers, style attributes. You get a scoring function that never requires touching the deployed system.
• Non-autoregressive generation is a controllable-text-generation substrate. This is the takeaway most orthogonal to safety. If you need constrained generation — grammar conformance, watermark evasion, keyword-constrained copy, program repair — a masked diffusion model gives you arbitrary-order revision under an arbitrary scoring function, which AR sampling structurally cannot. Anyone who has fought with beam search over an AR decoder should be paying attention to this framing regardless of their interest in jailbreaks.
• The defensive read is free and immediate. Their attack signal is a detector. A weighted safety-neuron activation score computed on the *input prompt, before generation, is a cheap screening layer with no extra forward pass. If their AUROC number is even half true out of distribution, that is a deployable filter — which is a slightly embarrassing gift for an attack paper to hand over.
• Cost-per-jailbreak is the metric that matters for threat modeling. Twenty offline surrogate episodes versus hundreds of live victim queries is not a marginal improvement; it removes the rate-limiting and abuse-detection defenses that most production systems actually rely on. If you build guardrails, budget for attackers who never query you before the attack.