DiaLLM: Understanding a Dialect Doesn't Mean You Can Speak One

Paper: 2607.07669 Authors: Jordan Painter, Dipankar Srirag, Adarsh Kappiyath, Diptesh Kanojia, Aditya Joshi, Lu Yin Categories: cs.CL, cs.AI

The Gap

Most work on dialectal NLP has focused on the comprehension side — can a model parse African American Vernacular English, Indian English, or Scouse without breaking? The answer, increasingly, is yes. Benchmarks like SAE-NLI and dialectal QA tasks show that continual pretraining and instruction tuning on dialectal data can close comprehension gaps. But nobody has seriously asked the reciprocal question: can these models write in a dialect, not just understand it?

The unstated assumption in the field has been that robustness (understanding) and generation (production) are two sides of the same coin — improve one and the other follows. Prior dialect adaptation papers measure success with classification-style benchmarks (NLI accuracy, NER F1, sentiment accuracy) that test comprehension, not generation quality. There’s no controlled comparison of how different training paradigms (continual pretraining, SFT, alignment) each contribute to dialectal generation specifically, across multiple dialects and model families simultaneously.

This paper argues that assumption is wrong, and proves it.

Problem: LLMs understand dialects but only produce standard English
    |
    v
Assumption in field: robustness ~ generation (improve one, get both)
    |
    v
Method: Controlled ablation across 3 model families x 3 dialects x
        3 training stages (CPT, SFT, alignment) with human eval
    |
    v
Evidence: Robustness improves with CPT+SFT; generation responds to
          alignment; but reward-optimal generation is NOT human-preferred
    |
    v
Conclusion: Robustness and generation are dissociated;
            reward design for dialectal gen needs rethinking

The Increment

One sentence: Before this paper, dialect adaptation was a comprehension problem measured by benchmarks; after this paper, we know it’s actually a generation problem where benchmarks systematically mislead, and the best reward-optimized dialectal output is not what humans actually prefer.

Core Mechanism

DiaLLM is not a single model — it’s a systematic experimental framework. The authors take three open-weight model families (Llama-3, Mistral, and Qwen-2.5, each at roughly 7B scale) and subject them to a controlled pipeline with three stages: continual pretraining (CPT), supervised fine-tuning (SFT), and alignment. Each stage is designed to isolate its contribution to dialectal generation.

The data comes from the International Corpus of English (ICE), which contains naturally occurring text in three varieties: Australian English (en-AU), Indian English (en-IN), and Northern British English (en-NB). For CPT, models are continually pretrained on ICE dialectal text. For SFT, they create instruction-following pairs from ICE data. For alignment, they train a reward model on dialectal preferences and apply three different strategies: (1) broad alignment (general helpfulness + dialect preference combined), (2) explicit variety-targeted alignment (reward specifically for producing the target dialect), and (3) a hybrid approach.

The key analytical move is what they measure. They run standard benchmarks (perplexity, dialectal NLI, etc.) but also conduct human evaluation where annotators judge whether outputs are dialectal and which outputs they prefer. This dual measurement is what reveals the dissociation.

ICE Corpus (en-AU, en-IN, en-NB)
    |
    +---> Continual Pretraining (CPT)
    |         |
    |         +---> measure: perplexity, comprehension benchmarks
    |         |
    |         v
    |     SFT on dialectal instruction pairs
    |         |
    |         +---> measure: benchmarks improve further
    |         |
    |         v
    |     Alignment (3 strategies)
    |         |
    |         +-- broad alignment
    |         +-- explicit variety-targeted
    |         +-- hybrid
    |         |
    |         v
    +---> Human Evaluation (dialectal? preferred?)
              |
              v
         Gap: benchmark-optimal != human-preferred

Think of it like training a chef. CPT is like having the chef apprentice in a regional kitchen — they absorb the vocabulary, rhythms, and ingredient combinations of that cuisine. SFT is like giving them recipe cards for regional dishes — now they can follow specific patterns. But alignment is like giving them a food critic’s scoring rubric. The paper finds that the chef who optimizes hardest for the critic’s score produces dishes that technically hit every criterion, but that actual diners find uncanny — too much “dialect flavor” in places where it feels forced. The diners prefer the chef who learned the cuisine organically and seasons naturally.

Key Concepts

• Dissociation of robustness and generation: Imagine you learn French by reading novels. You can now pass a French reading comprehension test — that’s robustness. But if someone asks you to *write a novel in French, you’ll default to textbook sentences, not literary prose — that’s the generation gap. This paper shows that for dialects, these two abilities develop through different training mechanisms and don’t transfer automatically. CPT and SFT drive comprehension gains; alignment is needed for generation, but benchmark metrics can’t see the difference.

• Reward-quality gap in dialectal generation: This is the paper’s sharpest finding. When you train a reward model to score “how dialectal is this text?” and then optimize a language model against that reward, the model learns to maximize the score by over-saturating dialectal markers — sprinkling dialect features everywhere regardless of whether a human would naturally produce them there. It’s like a student who learns that using big vocabulary words gets higher essay scores, so they stuff every sentence with thesaurus entries. The essay scores well on the rubric but reads terribly. The alignment strategy that most aggressively optimizes the dialectal reward produces text that human evaluators rate as less natural and less preferred.

• Explicit variety-targeted adaptation: Instead of training a general “be helpful and also dialectal” reward model, you train one that specifically says “produce Australian English” or “produce Indian English.” This turns out to be more effective at producing recognizable dialect output than broad alignment, which tries to do too many things at once. The analogy: a specialist vs. a generalist. The specialist knows exactly what features mark a dialect and targets them precisely.

Framework Shift

Before (mainstream approach):        After (this paper):

  Dialect NLP = Comprehension        Dialect NLP = Comprehension
       |                                   |         \
       v                                   v          \
  Benchmarks (NLI, NER, QA)         Benchmarks     Generation
       |                                   |          |    |
       v                                   v          v    v
  "Model understands dialect"       "Understands"  "Writes"  "Humans
                                                             prefer?"
                                              |          |
                                              v          v
                                         Not the same thing
                                         (dissociated)

From treating dialect adaptation as a comprehension benchmarking problem to recognizing it as a generation problem where human preference and automated reward diverge — the core shift is that good dialectal generation requires fundamentally different training signals than good dialectal understanding.

Expert Assessment

Problem choice: This is a genuine and underexplored gap. The field has been measuring dialect adaptation with comprehension benchmarks for years and implicitly assuming generation follows. The authors are right that this assumption has gone largely untested. The choice of three dialects (Australian, Indian, Northern British) is sensible — they’re typologically diverse, well-resourced in ICE, and represent different degrees of divergence from US English. This sits at the right point in the field’s trajectory: now that we can make models *understand dialects, the generation frontier is the natural next step.

Method maturity: The methodology is thorough but largely compositional — they’re combining known ingredients (CPT, SFT, RLHF-style alignment) rather than inventing new ones. The contribution is the systematic comparison, not algorithmic novelty. This is valuable science but not clever engineering. One concern: the reward model training for dialectal alignment is somewhat standard, and the paper’s own finding that this reward misaligns with human preference suggests the reward design itself is the bottleneck. They acknowledge this but don’t offer a fix, which feels like a limitation they could have pushed harder on.

Experimental integrity: The baselines are fair — they compare against base models and models with only CPT or only SFT, which isolates each stage’s contribution. The human evaluation setup with dialectal recognition + preference rating is well-designed. However, the three model families (Llama-3, Mistral, Qwen-2.5) are not perfectly controlled — they differ in pretraining data, architecture details, and scale, which the paper sometimes glosses over when making model-family-level claims. The finding that the reward-quality gap shows up “most clearly on two of the three families” is a bit convenient — the weaker family result could indicate the effect is less robust than claimed.

Writing quality: The abstract is excellent — dense and precise. The related work section is competent but could do more to position against specific prior claims. The weakest section is the linguistic analysis (Section 5), where they corroborate the reward-quality gap with independent linguistic features but the analysis feels rushed and the connection to prior dialectological work is thin. If they rewrote Section 5 with a proper linguistic framework (e.g., Biber’s multidimensional analysis or Labovian variation metrics), the paper would feel more interdisciplinary and less like an NLP paper paying lip service to linguistics.

Verdict: weak accept — The dissociation finding is real and important for the field’s direction, but the paper doesn’t go far enough in diagnosing *why rewards fail or proposing alternatives, which limits its actionable impact.

Takeaways

Three things a practitioner can steal:

1. Don’t trust comprehension benchmarks for generation tasks. If you’re adapting a model to produce a specific style, register, or dialect, your NLI/perplexity improvements may tell you nothing about output quality. Always pair automated metrics with human preference evaluation, especially for stylistic generation.

2. Explicit variety-targeting beats broad alignment for stylistic generation. If you want a model to produce a specific register, train a reward model specifically for that register rather than mixing it with general helpfulness objectives. The specialist reward signal is cleaner and the model responds more reliably.

3. Aggressive reward optimization is a warning sign, not a success signal. When your RLHF-trained model scores highest on the reward model but humans don’t prefer it, the problem is almost certainly reward misspecification. Before scaling up optimization, audit whether your reward captures what humans actually value. This applies well beyond dialects — to any domain where you’re using learned reward models for stylistic or preference optimization.