REDDIT: Fixing Timestamp Drift in ASR Without Forgetting — Replay-Based Distribution Editing

Paper: 2607.05364 Authors: Cheng-Kang Chou, Ming-To Chuang, Ke-Han Lu, Chan-Jan Hsu, Hung-yi Lee Categories: cs.CL, cs.AI, cs.SD

The Gap

Autoregressive ASR systems like Whisper have popularized timestamp-as-token decoding: the model emits special timestamp tokens interleaved with transcript tokens, producing aligned transcriptions without frame-level forced aligners or post-hoc heuristics. This is elegant and works well in normal speech. But there’s a catch nobody stress-tested rigorously: when the audio contains long non-speech spans — pauses, silence, music breaks — the autoregressive decoder loses track of where it is on the time axis. The text stays plausible, but the timestamps drift. A 10-second silence might get reported as 3 seconds, or a gap might stretch an entire segment forward.

Prior work largely sidestepped this. Frame-level aligners (CTC, attention-based) need architecture changes. Post-processing methods (VAD + interpolation) are brittle and disconnected from the model’s own representations. And the obvious fix — fine-tune on corrected timestamps — turns out to be a trap: naively supervised fine-tuning (SFT) on timestamp tokens catastrophically degrades non-timestamp behavior. On Whisper-tiny, naive SFT pushed Common Voice English MER from 41.3% to 524.2%. The model unlearns how to transcribe.

So the gap is precise: how do you correct a narrow slice of model behavior (timestamp accuracy over non-speech spans) without destroying everything else the model knows? Nobody had a principled answer. This paper provides one.

Problem
|
Long non-speech spans cause timestamp drift in
autoregressive ASR (Whisper et al.)
|
v
Prior attempts
|
+-- Frame-level aligners: need architecture redesign
+-- Post-processing (VAD + interpolation): brittle, disconnected
+-- Naive SFT on timestamps: catastrophic forgetting
|   (MER 41.3% --> 524.2% on Whisper-tiny)
|
v
Insight: fine-tuning corrupts the decoder distribution
because the teacher-forced prefix context shifts
|
v
REDDIT method
|
+-- Stage 1: edit timestamp targets under
|   replayed (frozen) base model decoder context
|   + match base distribution on non-timestamp tokens
+-- Stage 2: short edited-prefix refinement
|
v
Evidence
|
+-- long-gap mIoU: 38.7% --> 95.0%
+-- mixed-gap AAS: 2752ms --> 223ms
+-- CV-en MER preserved: 41.3% (vs 524.2% naive SFT)
+-- 34.9 hours targeted data, 1.6% params updated
|
v
Conclusion
|
Lightweight replay-based distribution editing
corrects narrow model behavior without forgetting

The Increment

One sentence: Before this paper, correcting timestamp drift in autoregressive ASR required either architectural changes or accepted catastrophic forgetting; after this paper, there exists a lightweight post-training recipe that fixes timestamps in 34.9 hours on 1.6% of parameters while preserving everything else.

Core Mechanism

REDDIT operates in two stages, both built on a single principle: when you want to change one token’s prediction but keep everything else intact, you should show the model its own “clean” context while only modifying the target of interest.

Stage 1 — Replay-Based Distribution Editing. The base (frozen) model and a trainable copy (edited model) both process the same input. The base model autoregressively generates decoder hidden states — these represent “what the model would have thought if we hadn’t touched it.” The edited model receives these replayed hidden states as context when predicting timestamp tokens. The training objective has two heads: for timestamp tokens, minimize the distance to corrected (ground-truth) timestamps; for non-timestamp tokens (the actual transcript), minimize KL divergence to the frozen base model’s output distribution. This ensures the edited model’s predictions on regular text tokens remain anchored to the original model’s behavior. Only a small adapter and the timestamp-prediction parameters are updated.

Stage 2 — Edited-Prefix Refinement. Stage 1 uses the base model’s context, which is “clean” but doesn’t reflect the edited model’s own corrected timestamps. Stage 2 closes this loop: the edited model now processes its own corrected timestamp predictions as prefix context for a short window, then predicts the next tokens. This is a brief refinement pass that teaches the model to be self-consistent — to handle its own corrected timestamps gracefully rather than relying on the base model’s (potentially drifted) context.

Stage 1: Replay-Based Distribution Editing

[Audio Input] --> [Encoder (frozen)] --> encoder_out
                                            |
                 +---------------------------+
                 |                           |
          [Base Decoder]              [Edited Decoder]
          (frozen, replayed)          (trainable adapter)
                 |                           |
          base_hidden_states -----> used as replay context
                 |                           |
          base_logits                 edited_logits
                 |                           |
          (reference)              +-- timestamp tokens: loss vs corrected ts
                                  +-- non-ts tokens: KL(base_logits || edited_logits)


Stage 2: Edited-Prefix Refinement

[Audio + corrected_ts_prefix] --> [Edited Decoder]
                                          |
                                  predict next tokens
                                  using own corrected context
                                          |
                                  short refinement loss

Metaphor: The Radio Frequency Analogy. Imagine you’re a radio engineer. Your receiver (the ASR model) is tuned to a station (the audio), and it normally works fine. But when the station goes silent for a while (non-speech span), your dial drifts — the frequency readout says one thing, but the actual signal is somewhere else.

Naive fine-tuning is like grabbing the dial and forcing it to the right frequency. It works for that moment, but you’ve disturbed the entire tuning mechanism — now the radio can’t pick up any station properly (catastrophic forgetting).

REDDIT’s Stage 1 is like having a second, identical radio (the base model) that keeps playing normally. You carefully adjust only the dial on the first radio while using the second radio’s audio output as your reference. You measure: “Is the dial reading right? Good. Is everything else sounding the same as the second radio? Good.” You only touch the dial (timestamp tokens) and leave all other controls (non-timestamp tokens) locked to match the reference radio’s settings.

Stage 2 is like now turning off the reference radio and checking: “Does my adjusted radio stay on frequency when it’s operating on its own?” You listen for a short while to confirm self-consistency, making tiny final adjustments if needed.

Without the reference radio (replayed base distribution), any adjustment you make to the dial would ripple through and change how the whole receiver behaves. The replay is what makes surgical correction possible.

Key Concepts

• Timestamp Drift: In autoregressive ASR, the model generates a sequence of tokens like <|0.00|> Hello <|1.20|> world <|2.40|> <|3.50|> ... <|4.20|> goodbye. Each <|...|> is a predicted timestamp. The model has learned to predict these as regular tokens, not through explicit alignment. When there’s a 5-second silence in the audio, the model must “imagine” that 5 seconds have passed — but it has no internal clock. It just generates whatever timestamp token seems plausible based on its training, and over long gaps, these estimates drift. Think of it like estimating seconds in your head while distracted: you might count to 10 but only 6 real seconds passed. The model makes a similar error, except it affects every downstream timestamp.

• Catastrophic Forgetting: Neural networks store knowledge in shared parameters. When you fine-tune on a new task (correct timestamps), gradient updates shift the same weights that encode transcription ability. The model doesn’t have separate “timestamp cells” and “transcription cells” — it’s all intertwined. So optimizing for timestamps can trash transcription. This is like repainting one wall of a house and accidentally cracking the plumbing that runs behind it. REDDIT’s KL-matching on non-timestamp tokens acts as a constraint that says “you can change the paint, but the plumbing must stay exactly as it was.”

• Distribution Editing via Replay: The core technical trick. Instead of training the edited model end-to-end (which would let corrections ripple through and corrupt the decoder state), you “freeze time” by feeding the base model’s hidden states into the edited model’s decoder. The edited model sees what the base model was thinking and only changes its prediction at timestamp tokens. This is surgical: you’re editing one line of the model’s “thought process” while keeping every other line identical. It works because the model’s representation of “what to say next” (transcription) and “when to say it” (timestamps) can be partially disentangled if you control the context carefully enough.

Framework Shift

Before (mainstream approach):
                                                    
  [Audio] --> [ASR Model] --> [Raw Timestamps]
                                    |
                              [Post-processing]
                              (VAD + interpolation)
                                    |
                              [Adjusted Timestamps]
                                    |
                              [Drift not fully fixed]
                              [Or: fine-tune = forget]


After (this paper):

  [Audio] --> [Encoder] --> encoder_out
                                 |
                  +--------------+--------------+
                  |                             |
           [Base Decoder]                [Trainable Adapter]
           (frozen)                      (1.6% params)
                  |                             |
           replay hidden states ------> context
                  |                             |
           base_dist (reference)    +-- correct timestamps
                                    +-- match base_dist on text
                                                |
                                          [Edited Model]
                                          (correct + preserved)

From post-hoc patching or destructive fine-tuning to surgical replay-based distribution editing, the core shift is treating timestamp correction as an isolated edit operation on the model’s generation process rather than a retraining problem.

Expert Assessment

Problem choice: Real gap, well-motivated. Timestamp drift over silence is something anyone who’s built on Whisper has probably noticed but tolerated. The 15-system benchmark (including modern audio-language models) makes the scope clear: this isn’t a Whisper-only issue, it’s architectural. The gap sits at the intersection of ASR alignment and continual learning — two active areas that rarely talk to each other.

Method maturity: Clever, not brute force. The replay-based editing idea is the real contribution — it’s a general principle (edit one behavior by anchoring to frozen reference distributions) that happens to be applied to timestamps here. The two-stage design (base-replayed then self-replayed) addresses a real subtlety: models need to learn to handle their own corrected outputs, not just the base model’s. That said, the VAD-based synthetic data construction is practical but somewhat brittle — it assumes non-speech gaps can be cleanly inserted, which may not reflect all real-world drift scenarios (think overlapping speakers, gradual noise transitions).

Experimental integrity: The numbers are striking but hold up to scrutiny. The MER preservation (41.3% vs 524.2%) is the key comparison and it’s dramatic. The 15-system evaluation on self-built benchmarks is thorough, though the benchmarks themselves aren’t released (as far as I can tell), which limits reproducibility. The ablation between Stage 1 and Stage 2 would strengthen the paper — we don’t see how much each stage contributes independently. The 34.9-hour data budget and 1.6% parameter update are compelling practical claims. One concern: all main results are on Whisper-tiny. How does this scale to Whisper-large or other architectures? The paper touches on this but doesn’t deep-dive.

Writing quality: Solid but not exceptional. The motivation section is clear. The method description could benefit from a cleaner mathematical formulation — the two objectives (timestamp correction loss and KL matching) are described in prose but a formal loss function in one place would help. The related work section is thin; there’s interesting adjacent work in knowledge distillation and model editing that deserves connection. Section 5 (the 15-system benchmark) is the paper’s strongest contribution to the community but reads as an afterthought — it should be elevated.

Verdict: weak accept — The core idea (replay-based distribution editing for surgical model correction) is transferable and the timestamp-drift problem is real, but the limited scale of experiments and missing ablations temper enthusiasm.

Takeaways

1. Replay-based distribution editing is a general-purpose tool. The principle — anchor to the frozen base model’s distribution on tokens you want to preserve, only supervise the tokens you want to change — applies to any narrow model correction task. Think: fixing hallucination on specific entity types, correcting bias in certain output categories, patching a model’s behavior on edge cases without full retraining.

2. KL matching on “don’t-touch” tokens is cheap insurance against forgetting. If you’re fine-tuning a model for any narrow behavioral change, adding a KL-divergence term that matches the original model’s distribution on everything except your target tokens costs almost nothing and prevents catastrophe. This is simpler than EWC, progressive nets, or other continual learning machinery.

3. Self-built stress-test benchmarks have outsized value. The authors’ 15-system evaluation on gap and long-gap audio is arguably more useful than their method. If you work with ASR timestamps, you now have a template for testing whether your system survives non-speech spans. Consider building your own version for your domain.