One Token Away from Collapse: How Simple Word Bans Break Instruction-Tuned LLMs

Paper: 2604.13006 Authors: Erfan Baghaei Potraghloo, Seyedarmin Azizi, Souvik Kundu, Massoud Pedram Categories: cs.CL, cs.AI

The Gap

Instruction-tuned models are supposed to be robust helpers. Prior work showed they handle format constraints (JSON, XML) reasonably well. But nobody checked what happens with trivial lexical constraints—like “don’t use the word ‘the’” or “avoid commas.” Turns out this is where the facade cracks.

The gap: We assumed instruction tuning makes models more capable across the board. This paper shows it actually creates a new fragility—models become dependent on specific surface forms to trigger their helpfulness behavior.

Problem: Instruction-tuned models seem robust
   |
   v
Assumption: Format constraints are the hard case
   |
   v
Method: Test trivial lexical constraints (ban one word/punctuation)
   |
   v
Evidence: 14-48% comprehensiveness loss, 77-100% baseline preference
   |
   v
Conclusion: Instruction tuning couples competence to narrow templates

The Increment

One sentence: Before this paper, we thought instruction tuning made models more robust; after, we know it creates a hidden dependency on surface-form templates that collapses under trivial constraints.

Core Mechanism

The paper tests instruction-tuned models (Llama, Mistral, Qwen, GPT-4o-mini) and their base versions with simple constraints: ban a punctuation mark (comma, period, colon) or common word (the, and, to). They measure response length and quality through pairwise comparison.

The collapse happens before generation starts. Linear probes trained on prompt representations (the model’s internal state after reading the constraint) predict final response length with R² = 0.51-0.93. This means the decision to produce a short, unhelpful response is encoded in how the model represents the constrained prompt—it’s a planning failure, not a generation failure.

Constrained Prompt --> [Model Representation] --> Planning Decision
                              |                         |
                              |                         v
                       Linear Probe                Short/Long?
                       Predicts Length              (R² = 0.51-0.93)
                       (R² = 0.93)
                              |
                              v
                       Collapse happens HERE
                       (before any tokens generated)

Two-pass generation confirms this: generate freely first, then rewrite with constraints. This recovers 59-96% of response length, proving the model has the knowledge but fails to plan its use under constraints.

Think of it like a chef who’s been trained to follow recipes exactly. Give them a recipe that says “make pasta but don’t use water,” and instead of adapting (use broth, steam it, whatever), they just… stop. They’ve learned “pasta = boiling water” as a rigid template, not a flexible concept. The instruction-tuned model learned “helpful response = uses commas and ‘the’” as a template. Ban those, and the template doesn’t fire, so helpfulness collapses. The base model (untrained chef) doesn’t have this template dependency—it just tries stuff, sometimes good, sometimes bad, but never systematically collapses.

Key Concepts

• Instruction Tuning Fragility: Instruction tuning teaches models to produce helpful, structured responses by showing them examples. But it does this by creating internal templates—patterns like “helpful answer = markdown formatting + common words + punctuation.” When you break the template (ban a word), the model doesn’t adapt; it fails to recognize the task as “be helpful” and produces minimal output. Base models don’t have these templates, so they don’t collapse—they’re inconsistent but not systematically fragile.

• Planning vs Generation Failure: The model decides how long/comprehensive its response will be before generating any tokens. You can predict the final response length from the prompt representation alone (R² up to 0.93). This means the collapse isn’t “I tried to write a good response but couldn’t avoid the banned word”—it’s “I decided upfront to write a short response because the constraint broke my helpfulness template.” Two-pass generation (write freely, then rewrite) works because the first pass doesn’t have the constraint in the prompt representation, so planning succeeds.

• Pairwise vs Independent Evaluation: Standard LLM-as-judge evaluation (rate this response 1-10) detects only 3.5% quality drop. Pairwise comparison (which is better: constrained or baseline?) reveals 23% drop and 77-100% baseline preference. Why? Independent scoring is anchored to “is this response acceptable?” (yes/no threshold), while pairwise forces relative judgment. The constrained responses are often acceptable in isolation but clearly worse when compared directly—a methodological blind spot in how we assess constrained generation.

Framework Shift

Before (assumed):                    After (this paper):

Instruction Tuning                   Instruction Tuning
       |                                    |
       v                                    v
  More Robust                          Template Dependency
  More Capable                              |
       |                                    +-- Helpful (template matches)
       v                                    |
  Handles Constraints                      +-- Collapse (template breaks)
  (format, lexical, etc.)
                                      Base Model: No Template
                                           |
                                           v
                                      Inconsistent but Not Fragile

From “instruction tuning = general robustness” to “instruction tuning = template-dependent competence that collapses when templates break.”

Expert Assessment

Problem choice: Real gap. Prior work focused on format constraints (JSON, XML) where models do okay. Nobody checked trivial lexical constraints because they seemed too easy. Turns out “too easy” is where the interesting failure mode lives. The problem sits at the intersection of alignment research and robustness—timely and underappreciated.

Method maturity: Elegant simplicity. The constraints are almost comically simple (ban “the”), which makes the 48% comprehensiveness loss hit harder. The mechanistic analysis (linear probes, two-pass generation) is solid and goes beyond “here’s a weird failure mode” to “here’s why it happens.” Could’ve tested more constraint types (semantic constraints? length limits?), but the lexical focus is sufficient to make the point.

Experimental integrity: Baselines are fair. Testing both base and instruction-tuned versions of the same models is the right control. The pairwise evaluation (1,920 comparisons) is thorough. One concern: GPT-4o-mini judges its own constrained outputs—potential bias, though the effect is so strong (99% baseline preference) that bias seems unlikely to explain it. The MT-Bench replication across eight task categories strengthens the claim of generality.

Writing quality: The abstract and intro are sharp. The mechanistic analysis section (linear probes, two-pass generation) could be clearer—it’s dense and assumes familiarity with probing methods. The discussion of why base models don’t collapse is buried in the results; pulling it forward would strengthen the narrative. The pairwise vs independent evaluation insight is gold but feels like an afterthought—deserves more prominence.

Verdict: Strong accept — Identifies a real, underappreciated fragility in instruction-tuned models with clean experiments and actionable mechanistic insights.

Takeaways

For practitioners: If you’re deploying instruction-tuned models with any kind of constraint (content filters, style requirements, domain-specific rules), test with trivial lexical constraints first. If the model collapses when you ban “the,” your production constraints might trigger the same planning failure. Two-pass generation (generate freely, then rewrite) is a practical workaround that recovers most quality.

For researchers: Pairwise evaluation catches quality drops that independent scoring misses—use it for constrained generation tasks. The linear probe result (predicting response length from prompt representations) is a transferable technique for diagnosing planning failures in other contexts.

For alignment work: Instruction tuning creates template dependencies that aren’t visible in standard benchmarks. If you’re training models to be helpful, test whether helpfulness survives when you break surface-form patterns. The base model comparison is a useful control—if base models don’t show the effect, instruction tuning created it.