Natural Ungrokking: Asymmetric Control of Which Rules Survive Pretraining

Paper: 2606.26050 Authors: Juliana Li, Diya Sreedhar Categories: cs.LG, cond-mat.dis-nn, cs.AI, cs.CL

The Gap

Existing work on “grokking” — sudden generalization after overfitting — focuses on late-stage rule emergence, often in controlled synthetic settings. Prior work assumes that once a rule is learned during pretraining, it either persists or degrades only through catastrophic forgetting (explicit data deletion). The specific boundary here: *rules can disappear during pretraining, not just after it, without any change to the training data’s evidence for that rule. The loss curve shows no signal. No prior work characterizes this within-run reversal or explains which rules survive based on corpus statistics alone.

[Problem: models forget learned rules mid-pretraining]
        |
[Assumption: forgetting requires data deletion or signal in loss]
        |
[Gap: rule disappears *while evidence stays*; loss is flat]
        |
[Method: measure "support frequency" — how often rule wins in training stream]
        |
[Evidence: support frequency predicts survival across corpora/seeds/budgets]
        |
[Conclusion: a simple corpus statistic, not model arch, decides rule fate]

The Increment

One sentence: Before this paper, we didn’t know that a model’s decision to keep or drop a learned rule during pretraining is predictable from a single corpus statistic; after this paper, we can forecast rule survival and asymmetrically control it — destroy but not restore.

Core Mechanism

The method is not a new algorithm — it’s an experimental framework for measuring and controlling the natural emergence and collapse of simple rules during autoregressive language model pretraining. The core components are: (1) a training corpus structured with two competing patterns (a target rule like “female name -> she” and a higher-frequency surface alternative like “verb -> it”), (2) a metric called support frequency: the count of training examples where the rule “wins” (the correct next-token matches the rule’s prediction), and (3) a behavioral probe evaluating whether the model follows the rule on held-out inputs.

The data flow is: Corpus statistics → support frequency → log-probability margin between rule and competitor → behavioral probe score. The key operation is measuring at which support frequency the margin crosses zero (i.e., when the competitor becomes more probable than the rule). This zero-crossing precedes behavioral collapse by ~100 steps. The paper shows that across two corpora, three compute budgets, and three random seeds, the critical support frequency for survival is consistent. In a second experiment, they perform “in-place editing”: flipping supports to counter-evidence kills a rule with monotonic dose-response; flipping them back, even to absurdly high levels (450x natural survival level), cannot restore it.

[Corpus: two competing patterns]
        |
   [Count: support frequency = rule wins / total occurrences]
        |
   [Measure: log-probability(rule) - log-probability(competitor)]
        |
   [Predict: if margin cross zero? -> rule collapses in ~100 steps]
        |
   [Control: flip supports to counter-evidence -> monotonic kill]
        |
   [Control: flip back to supports -> NO recovery (asymmetric)]

Structural metaphor: A tide pool with two species of barnacles. The pool is the model’s internal representation of the pronoun-gender rule. Two barnacle species compete for space on a single rock: species R (the “rule” — female name -> she) and species C (the “competitor” — verb -> it). The “support frequency” is how often the daily tide brings nutrients specifically to species R. The “log-probability margin” is the visible boundary between the two colonies. When the tide shifts (support frequency drops below a threshold), the C colony overgrows the R colony completely within about 100 high tides — the boundary crosses zero. Now here’s the twist: if you *remove nutrients from R (flip to counter-evidence), the R colony shrinks reliably; but if you dump 450 times the natural nutrient level back onto R’s spot, the C colony does not retreat. The rock face is permanently claimed by C. The model doesn’t “forget” R — it gets displaced, and the displacement is one-way. That’s the asymmetry.

Key Concepts

• Natural Ungrokking: The phenomenon where a language model learns a rule during pretraining, performs well on it, then spontaneously *reverses to near-zero performance — all while the training data still contains the same evidence for the rule. Unlike grokking (late emergence after overfitting), this is a within-run disappearance. The loss curve doesn’t blink. Example: at step 925, “Sue cried because ___” yields “she” with 0.94 accuracy; by step 3,500, same probe scores near zero, even though “she” still appears after female names in training data.

• Support Frequency: A simple corpus statistic: count of training examples where a specific rule “wins” — i.e., where the correct next token matches the rule’s prediction (e.g., after a female name, the target token is “she”). This is NOT raw frequency of the rule’s trigger, but frequency of the *correct outcome matching the rule. The paper shows this single number (not ratio, not model size, not parameter count) is the decisive variable for whether a rule survives. Threshold: if support frequency falls below ~0.2% of tokens (exact number corpus-dependent), the rule collapses.

• Asymmetric Control: The same experimental manipulation — flipping context-token pairs in the corpus — has opposite effects depending on direction. Flipping from supporting the rule to contradicting it (e.g., changing “Sue cried because she” to “Sue cried because he”) kills the rule with a clear dose-response: more flips → lower final probe score. But flipping from neutral/counter to supporting, even at 450x the natural support frequency, yields zero recovery. The model’s internal state is path-dependent: once the competitor displaces the rule, no amount of reinforcement re-instates it.

Framework Shift

Before (mainstream view):              After (this paper):
[Pretraining step 0 -> learn rule     [Pretraining step 0 -> learn rule
 -> rule persists unless              -> support frequency decides
 data explicitly removed]              survival; displacement is
                                       one-way]
[Loss curve: source of truth]         [Loss curve: blind to rule death]
[Forgetting: catastrophic,            [Forgetting: natural, continuous,
 event-driven]                         frequency-driven]
[Control: symmetric (add/remove)      [Control: asymmetric (destroy only,
 data -> rule appears/disappears]      cannot restore)]

One sentence: From “rules are learned properties of model + data” to “rules are fragile demographic outcomes of corpus ecology, with irreversibility.”

Expert Assessment

Problem choice: Real gap. The “grokking” literature studies sudden *emergence, but this paper studies sudden disappearance during pretraining — a distinct phenomenon with different mechanisms. The problem is non-obvious (you’d think rules learned from evidence stay learned) and empirically clean.

Method maturity: Clever, not brute force. The experimental design is minimal but precise: synthetic corpora that isolate the variable of interest, pre-registered thresholds, no architectural hacking. One could argue for simpler approaches (e.g., just measuring token statistics), but the paper’s core insight *is that this simple statistic predicts behavior — which is the point.

Experimental integrity: High. Pre-registration of thresholds before reading data is the gold standard for avoiding p-hacking. Baselines include multiple seeds, budgets, and corpora. The Pythia checkpoint replication adds external validity. No red flags. The asymmetric control experiment is particularly clean: same manipulation, opposite results, monotonic dose-response on one side, flat zero on the other.

Writing quality: Well-structured but dense. The most under-explained section is the “in-place editing” experimental setup — the distinction between “flipping support to counter-evidence” and “injecting support back” is the crux of the asymmetry result, yet the mechanics (how exactly they edit the corpus) are glossed over in a paragraph. Rewriting this section with a concrete example corpus snippet would elevate the paper significantly.

Verdict: weak accept — clean, reproducible result with a surprising and well-demonstrated asymmetry, though limited scope (simple rules, tiny models).

Takeaways

Practitioners steal these specifics: (1) Monitor rule-specific probe scores, not just loss — loss is blind to rule-level forgetting. (2) Use support frequency as a diagnostic lens before training to flag rules likely to self-destruct mid-run. (3) If you want to permanently remove a behavior, know that adding counter-evidence works once, but cannot be reversed by adding more supporting data — the asymmetry is a design constraint, not a bug. This directly impacts data augmentation strategies for safety: you can “brainwash” a model by flipping a few examples, but the model won’t recover if you later add the correct examples back. Explicitly: for content moderation fine-tuning, drop the original supportive data forever; don’t count on re-adding it later.