Tokenizer Transplantation: Making English ASR Models Speak Bengali

Paper: 2607.09598 Authors: Sanjid Hasan, Md. Abdur Rahman Categories: cs.CL

The Gap

Edge-efficient ASR models like Moonshine are heavily optimized for English. They use byte-level tokenizers (like ByT5’s approach) that work beautifully for Latin scripts — one or two bytes per character, predictable decoding. But Bengali is a different beast: it’s an abugida script with conjunct consonants, vowel signs, and complex grapheme clusters. When you feed Bengali through a byte-level tokenizer trained on English assumptions, words explode into chains of 9+ bytes on average. The decoder, trained to emit one token per meaningful unit, suddenly faces sequences 8-9x longer than expected. This triggers autoregressive collapse — the model degenerates into repeating tokens or producing garbage, because it was never trained to handle such long decoding chains.

Prior work either (a) trains models from scratch on Bengali (expensive, requires massive data), (b) uses multilingual models that are too heavy for edge deployment, or (c) fine-tunes but keeps the broken tokenizer, treating the symptom without fixing the cause. Nobody asked: what if we just swap the tokenizer’s vocabulary wholesale?

English-centric            Bengali audio input         Byte-level tokenizer
byte-level ASR     +-----> "আমার সোনার বাংলা"  +-----> 9.16 bytes/word avg
on Bengali                 (morphologically rich)      fragments everything
                                                      |
                                                      v
                                                 Autoregressive collapse
                                                 (sequence too long,
                                                  model degenerates)
                                                      |
                                                      v
                                                 Garbage output / loops

Proposed fix:  Transplant BanglaBERT WordPiece vocab
               + resize embedding layer
               + keep encoder weights frozen
               |
               v
               1.30 tokens/word avg
               85.8% shorter sequences
               |
               v
               Stable decoding, 21.54% WER

The Increment

One sentence: Before this paper, adapting a compact English ASR model to Bengali meant expensive retraining or accepting broken behavior; after, a surgical vocabulary transplant achieves competitive accuracy with zero pre-training cost.

Core Mechanism

The method has three stages: extraction, transplantation, and alignment.

Stage 1 — Extract. The authors pull the WordPiece vocabulary from BanglaBERT, a Bengali-specific masked language model. This vocabulary contains ~30K subword units that natively represent Bengali morphemes, common conjuncts, and word stems. It’s the “Bengali dictionary” the target model never had.

Stage 2 — Transplant. The decoder’s vocabulary is replaced wholesale. The token embedding matrix is resized from the original English byte-vocab dimensions to match BanglaBERT’s vocabulary size. New embedding vectors are initialized (the paper doesn’t specify the exact initialization, which is a minor gap). The decoder head is similarly resized to output logits over the new vocabulary.

Stage 3 — Fine-tune. Only the decoder and embedding layers are trained on Bengali speech data (the Lipi-Ghor dataset, 882 hours). The encoder — which handles acoustic features — can be frozen or lightly fine-tuned, since acoustic representation is more script-agnostic.

BanglaBERT           Moonshine ASR
(Bengali LM)         (English ASR)
     |                     |
     v                     v
[WordPiece Vocab]    [Byte-level Vocab]
  ~30K tokens         256 bytes
     |                     |
     +--------+  +---------+
              |  |
              v  v
         [Transplant]
         replace vocab
         resize embeddings
         resize decoder head
              |
              v
     [Fine-tune decoder on]
     [Bengali speech data]
              |
              v
     Bengali-capable
     edge ASR model

The structural metaphor: Organ transplant surgery.

Think of the ASR model as a patient whose brain (encoder) processes sound just fine, but whose mouth and tongue (decoder + tokenizer) only know how to speak English. When Bengali audio comes in, the brain hears it correctly, but the mouth tries to produce English phonemes — it stutters, loops, and collapses.

The paper’s approach is like a tongue transplant. You take a healthy Bengali tongue from a cadaver (BanglaBERT’s vocabulary), surgically swap it into the patient (resize embeddings, replace decoder vocab), then spend weeks in rehab (fine-tune on Bengali speech). The brain doesn’t need replacement — it was hearing sounds correctly all along. The critical insight is that you don’t need to rebuild the whole patient; you just need the right tongue.

The “anti-rejection drugs” in this analogy are the fine-tuning steps — you can’t just swap and walk away. The decoder needs to learn to coordinate with the new vocabulary, mapping Bengali subword tokens to acoustic patterns. But because the encoder is preserved, the rehabilitation is short and cheap.

Key Concepts

• Token Fertility: Imagine reading a book where every word is split into individual letters with spaces between them. “Hello” becomes “H e l l o” — five tokens instead of one. That’s what happens when Bengali meets an English byte tokenizer. Fertility measures how many tokens each original word gets split into. Bengali through English bytes: 9.16 tokens per word. Bengali through Bengali WordPiece: 1.30 tokens per word. Lower fertility = shorter sequences = faster, more stable decoding. It’s the single most diagnostic metric in this paper.

• Autoregressive Collapse: An autoregressive model generates one token at a time, feeding each output back as input for the next step. When sequences are much longer than training data ever showed, small errors compound. The model might start repeating the same token, or produce increasingly unlikely tokens until output is meaningless. Think of it like a game of telephone — the longer the chain, the worse the signal degradation. By shortening sequences 85.8%, the paper sidesteps this failure mode entirely.

Framework Shift

Before (mainstream approach):          After (this paper):

  Bengali audio                        Bengali audio
       |                                    |
       v                                    v
  [Fixed English tokenizer]            [Bengali WordPiece vocab]
       |                                    |
       v                                    v
  Long byte sequences                  Short subword sequences
  (9x expected length)                 (1.3 tokens/word)
       |                                    |
       v                                    v
  [Decoder trained on                  [Decoder fine-tuned on
   English sequences]                   Bengali sequences]
       |                                    |
       v                                    v
  Autoregressive collapse              Stable generation
  / garbage output                     21.54% WER

  Core assumption:                     Core assumption:
  Tokenizer is architecture,           Tokenizer is a component,
  not a replaceable part               swappable like a plug

From treating the tokenizer as a fixed architectural commitment to treating it as a swappable component, the core shift is vocabulary modularity — the idea that you can decouple acoustic processing from script-specific tokenization.

Expert Assessment

Problem choice: This is a genuine, underexplored gap. Most edge ASR work focuses on English and a handful of high-resource languages. Bengali has 230M+ speakers and is structurally different enough from English to expose real failure modes. The identification of byte-level tokenization as the root cause (not just “model too small” or “not enough data”) is sharp and well-motivated.

Method maturity: It’s a clever surgical insight executed with straightforward engineering. The “transplant” is not novel in isolation — people have swapped tokenizers before in NLP — but applying it specifically to fix autoregressive collapse in edge ASR is the contribution. One concern: the paper doesn’t deeply explore initialization strategies for the new embeddings. Random initialization works, but would better initialization (e.g., from acoustic-semantic alignment) help more? Also, a simpler baseline like just using a multilingual SentencePiece vocab might have been worth comparing.

Experimental integrity: The 882-hour Lipi-Ghor dataset is a reasonable scale. The WER of 21.54% is competitive but not SOTA for Bengali ASR in absolute terms — though the point is edge efficiency, not beating cloud models. The RTF of 0.0053 is convincingly fast. Main flag: no ablation on how much the encoder needs retraining. Is the encoder truly plug-and-play, or does it need more work than claimed? A frozen-encoder vs. fine-tuned-encoder comparison would strengthen the paper.

Writing quality: The paper is clear but brief to a fault. Section 3 (method) reads more like a recipe than an explanation — you can follow the steps but don’t always understand *why each choice was made. The related work section is thin; it doesn’t engage with the broader tokenizer adaptation literature from machine translation (where subword segmentation has been debated for years). A deeper discussion of why WordPiece specifically (vs. BPE, SentencePiece, unigram) would elevate the framing significantly.

Verdict: weak accept — A clean, practical contribution with a clear problem-solution fit, but underexplored in depth and missing key ablations that would make it a strong paper.

Takeaways

1. Diagnose before fixing. The paper’s real contribution is the diagnosis — byte-level tokenizer fertility as root cause of collapse — not just the fix. Practitioners working on multilingual models should measure token fertility per language *before assuming the model architecture is the bottleneck.

2. Vocabulary is a component, not architecture. Treat your tokenizer’s vocabulary as a swappable module. If you’re adapting a model to a new language/script, consider whether a vocabulary transplant (from a pretrained LM in the target language) could be cheaper than full retraining.

3. Sequence length is a hidden cost multiplier. In autoregressive models, doubling sequence length doesn’t just double compute — it compounds decoding instability. Any preprocessing that shortens sequences (better tokenization, compression, summarization) has outsized effects on reliability.

4. Encoder-decoder models have asymmetric script sensitivity. Encoders handling acoustic features are relatively script-agnostic. Decoders generating tokens are deeply script-coupled. This asymmetry means you can often freeze the encoder and only swap/adapt the decoder side — a much cheaper operation.