TildeOpen LLM: Teaching a 30B Model to Speak 34 European Languages Fairly

Paper: 2603.08182 Authors: Toms Bergmanis, Martins Kronis, Ingus Jānis Pretkalniņš, Dāvis Nicmanis, Jeļizaveta Jeļinska, Roberts Rozis, Rinalds Vīksna, Mārcis Pinnis Categories: cs.CL, cs.AI

The Gap

Existing multilingual LLMs like mGPT, BLOOM, and Llama follow the natural distribution of web data during training. This sounds reasonable until you realize that “natural” means English gets 90% of the attention while Estonian gets crumbs. Models like BLOOM tried upsampling low-resource languages, but they still trained on the upsampled distribution continuously — imagine force-feeding someone Estonian text for months straight. The result? Baltic and Finno-Ugric languages remain second-class citizens with error rates 10x higher than English.

The core assumption here is that data imbalance isn’t just about quantity — it’s about when and how you expose the model to different languages during training. Prior work treated this as a static mixing problem. This paper asks: what if we alternate between giving the model equal exposure to all languages and letting it see the natural web distribution?

Problem: English dominates → Low-resource languages suffer
    |
    v
Assumption: Training schedule matters as much as data volume
    |
    v
Method: Curriculum learning (alternate uniform + natural distribution)
    |
    v
Evidence: 10x error reduction in Baltic/Finno-Ugric languages
    |
    v
Conclusion: Smart scheduling beats naive upsampling

The Increment

One sentence: Before this paper, multilingual LLMs either followed natural data distributions (leaving small languages behind) or upsampled uniformly (wasting compute on redundant exposure); after, we have a training recipe that alternates between equal and natural exposure, achieving better low-resource performance without extra compute.

Core Mechanism

The method has three components working in sequence. First, dataset upsampling creates two parallel data streams: one where all 34 languages get equal representation (uniform distribution), and one that mirrors the natural web (English-heavy). Second, a curriculum scheduler alternates between these streams during training — think of it as switching between two different textbooks every few chapters. Third, the model trains on 30B parameters using standard transformer architecture, but the magic is in what it sees and when.

Here’s the data flow: raw multilingual corpora → upsample low-resource languages → split into uniform stream and natural stream → curriculum scheduler decides which stream to sample from at each training step → model updates weights → repeat for the full training run. The operations are straightforward: no architectural changes, no fancy loss functions, just controlled exposure timing.

Raw Data (34 languages, imbalanced)
    |
    v
+-------------------+
| Upsampling Engine |
+-------------------+
    |
    +---> Uniform Stream (all languages equal)
    |
    +---> Natural Stream (web distribution)
    |
    v
+-------------------+
| Curriculum        |  <-- Alternates between streams
| Scheduler         |
+-------------------+
    |
    v
+-------------------+
| 30B Transformer   |
| (standard arch)   |
+-------------------+

Think of this like training a chef who needs to master both everyday cooking and exotic cuisines. Most culinary schools (standard training) just throw you into a kitchen where 90% of orders are burgers — you get really good at burgers but panic when someone orders Georgian khachapuri. TildeOpen’s approach is like a rotation system: Week 1, you cook equal amounts of every cuisine (uniform stream) so you don’t forget the rare dishes. Week 2, you work the real kitchen with its burger-heavy orders (natural stream) so you learn what actually matters in production. Alternate these rotations, and you end up fluent in both common and rare dishes without spending extra time in school.

The uniform stream is your “don’t forget the rare stuff” rotation. The natural stream is your “learn what matters in the real world” rotation. The curriculum scheduler is the head chef deciding when to switch your station. The model’s weights are your muscle memory that improves with each rotation.

Key Concepts

• Curriculum Learning in Language Training: Imagine teaching a kid to read. You don’t hand them Tolstoy on day one — you start with picture books, then chapter books, then novels. Curriculum learning applies this to model training: control the order and mixture of what the model sees. In this paper, it’s not about difficulty progression but *distribution progression. The model alternates between seeing all languages equally (so it doesn’t forget Estonian exists) and seeing languages as they naturally occur on the web (so it learns that English matters more in practice). The curriculum isn’t “easy to hard” — it’s “balanced to realistic” and back again, repeatedly. Concrete example: In phase 1, the model might see 100 Estonian sentences and 100 English sentences. In phase 2, it sees 10 Estonian and 1000 English. Phase 3 goes back to 100-100. This prevents catastrophic forgetting of rare languages while still teaching real-world distributions.

• Upsampling vs. Curriculum Scheduling: Upsampling means copying rare data to match common data — if you have 1000 English sentences and 10 Estonian ones, you repeat those 10 Estonian sentences 100 times. Simple, but wasteful: the model sees the same Estonian sentences over and over. Curriculum scheduling is smarter: you create two separate data streams (one upsampled to uniform, one natural), then alternate between them during training. It’s the difference between eating the same meal 100 times versus eating it once per week for 100 weeks while varying your other meals. The model gets repeated exposure without the redundancy problem, because the *context around each exposure changes as the model evolves.

Framework Shift

Before (mainstream approach):        After (this paper):

Training Data (fixed mixture)        Training Data (two streams)
    |                                     |
    v                                     +---> Uniform
[EN EN EN EN EN ET]                      |     [EN ET LV ...]
[EN EN EN EN EN ET]                      |
[EN EN EN EN EN ET]                      +---> Natural
    |                                           [EN EN EN ET]
    v                                     |
Model sees same                           v
distribution forever                  Scheduler alternates
                                          |
                                          v
                                      Model sees both
                                      distributions cyclically

From static mixing to dynamic alternation, the core shift is treating training distribution as a time-varying curriculum rather than a fixed recipe.

Expert Assessment

Problem choice: This is a real gap, not manufactured. The linguistic equity issue in LLMs is well-documented, and existing solutions (naive upsampling, larger models) are either ineffective or expensive. The problem sits at a sweet spot: important enough to matter for millions of speakers, tractable enough to solve without inventing new architectures. It’s incremental but addresses a genuine pain point.

Method maturity: This is clever reuse of known techniques (curriculum learning, upsampling) rather than a novel algorithm. The insight is in the *combination — alternating between distributions instead of blending them once. That said, the paper doesn’t deeply explore why this works beyond “it prevents forgetting.” A more mature treatment would include ablations on alternation frequency, phase duration, and transition strategies. The approach feels like it landed on a good heuristic without fully understanding the mechanism.

Experimental integrity: Baselines are fair — they compare against BLOOM, mGPT, and other open multilingual models. The 10x error reduction claim is backed by human evaluation, which is expensive and credible. However, the paper is light on failure cases. What happens to high-resource languages? Do they suffer from the uniform phases? The benchmarks are standard (multilingual comprehension, generation), but I’d want to see more stress tests: code-switching, low-frequency words, domain transfer. The compute efficiency claim (fewer resources than competitors) is mentioned but not quantified precisely — how much fewer?

Writing quality: The abstract and intro are strong, but the method section is too high-level. The curriculum schedule details are vague — how long is each phase? What triggers the switch? The results section dumps numbers without enough analysis of *why certain languages improved more than others. If they rewrote Section 4 (Method) with concrete hyperparameters and decision rationale, the paper would jump from “interesting idea” to “reproducible recipe.”

Verdict: weak accept — Solid contribution to an important problem with credible results, but the method lacks depth and the writing could be tighter.

Takeaways

The transferable idea here is distribution alternation as a regularization technique. If you’re training on imbalanced data (which is most real-world scenarios), don’t just upsample and blend — create multiple views of your data with different balance properties, then alternate between them during training. This applies beyond languages: imagine training a medical diagnosis model where rare diseases get “uniform phases” to prevent forgetting, alternating with “natural phases” that reflect actual hospital admission rates. Or a recommendation system that alternates between showing equal exposure to niche content and popular content.

The specific technique to steal: maintain two data loaders (balanced and natural), switch between them every N steps, and track performance on rare classes separately. The curriculum doesn’t need to be sophisticated — even a simple alternation schedule can prevent the model from collapsing into “just predict the majority class.”

One warning: this only works if your rare classes have some data. If you have 5 examples of Estonian, no amount of curriculum magic will help — you’re still just memorizing those 5 examples. The method shines when you have hundreds or thousands of examples per rare class, enough to learn patterns but not enough to compete with the majority class in a naive mixture.