UniLID: Tokenizer-Based Language Identification for Low-Resource and Dialect Settings

Paper: 2602.17655 Authors: Clara Meister, Ahmetcan Yavuz, Pietro Lesci, Tiago Pimentel Categories: cs.CL

Abstract

Language Identification (LID) remains a critical yet challenging task in multilingual NLP, particularly for low-resource languages and closely related dialects. This paper introduces UniLID, an innovative approach that repurposes the UnigramLM tokenization algorithm for language identification. By learning language-conditional unigram distributions over a shared vocabulary while treating segmentation as language-specific, UniLID achieves remarkable sample efficiency—exceeding 70% accuracy with merely five labeled examples per language. The method demonstrates competitive performance against established baselines like fastText, GlotLID, and CLD3 on standard benchmarks, while offering substantial improvements in low-resource scenarios and fine-grained dialect distinction.

Key Contributions

• Tokenizer-as-classifier paradigm: Reframes LID as a probabilistic segmentation problem using UnigramLM’s expectation-maximization framework
• Extreme sample efficiency: Achieves over 70% accuracy with just 5 training samples per language, dramatically outperforming existing methods in few-shot settings
• Incremental language support: Enables adding new languages without retraining existing models, supporting modular and scalable deployment
• Dialect-aware identification: Delivers significant performance gains on fine-grained dialect identification tasks where traditional methods struggle
• Seamless integration: Naturally fits into existing tokenization pipelines, reducing computational overhead in multilingual LM workflows

Methodology: Probabilistic Segmentation for LID

UniLID builds on the UnigramLM tokenization algorithm, which models text segmentation as a probabilistic process. For a given text $x$, the algorithm finds the optimal segmentation $s^*$ by maximizing:

$s^* = \arg\max_s P(s|x) = \arg\max_s \prod_{t \in s} P(t)$

where $t$ represents individual tokens in the segmentation. UniLID extends this by introducing language-conditional distributions:

$P(x|L) = \sum_{s \in \text{Seg}(x)} \prod_{t \in s} P(t|L)$

The key insight is treating segmentation preferences as language-specific signals. Languages exhibit distinct tokenization patterns—for instance, agglutinative languages favor longer morpheme-based tokens, while isolating languages prefer character-level segmentation. By learning separate unigram distributions $P(t|L)$ for each language $L$ over a shared vocabulary, UniLID captures these linguistic preferences.

Parameter estimation uses the standard EM algorithm from UnigramLM, but applied independently per language. During inference, the model computes the likelihood of the input text under each language model and selects the language with maximum probability. This formulation is computationally efficient, requiring only forward passes through unigram distributions without neural network inference.

Experimental Results and Analysis

The authors evaluate UniLID across multiple dimensions, comparing against three strong baselines: fastText (neural n-gram classifier), GlotLID (transformer-based), and CLD3 (character n-gram with Naive Bayes).

Standard benchmarks: On high-resource test sets, UniLID achieves competitive performance with established methods, demonstrating that the tokenizer-based approach doesn’t sacrifice accuracy for efficiency. The method performs particularly well on languages with distinctive orthographic or morphological characteristics.

Low-resource superiority: The most striking results emerge in few-shot scenarios. With only 5 training samples per language, UniLID surpasses 70% accuracy, while neural baselines like fastText struggle to exceed 40%. At 10 samples, the gap widens further, with UniLID approaching 80% accuracy. This dramatic improvement stems from the method’s inductive bias—unigram distributions require far fewer parameters than neural models, making them less prone to overfitting on limited data.

Dialect identification: UniLID shows substantial gains on fine-grained dialect tasks, such as distinguishing between Arabic varieties or Chinese dialects. Traditional LID systems often collapse closely related varieties into single categories, but UniLID’s segmentation-based approach captures subtle lexical and morphological differences. For instance, dialectal variations in compound formation or affix usage produce distinct tokenization patterns that the model exploits.

Computational efficiency: Training time scales linearly with vocabulary size and corpus size, making UniLID orders of magnitude faster than neural approaches. The method also supports incremental updates—adding a new language requires only estimating its unigram distribution, leaving existing language models untouched.

Implications for Multilingual NLP

UniLID’s design philosophy challenges the prevailing trend toward ever-larger neural models for every NLP task. By demonstrating that classical probabilistic methods can outperform neural approaches in data-constrained settings, the work highlights the importance of inductive biases aligned with task structure.

The incremental learning capability addresses a practical pain point in production systems: as new languages or dialects emerge in user data, systems can adapt without expensive retraining. This modularity is particularly valuable for platforms serving diverse linguistic communities.

Integration with tokenization pipelines offers architectural elegance. Since modern LLMs already employ tokenizers, incorporating LID at this stage adds minimal overhead while providing early language detection for downstream routing, prompt engineering, or model selection. This could enable more efficient multilingual serving architectures where language-specific models are invoked only when needed.

The success on dialect identification opens possibilities for sociolinguistic applications, such as analyzing code-switching patterns, tracking language variation in social media, or building more inclusive language technologies that recognize non-standard varieties.

Takeaways

1. Tokenization algorithms encode rich linguistic information that can be repurposed for classification tasks beyond segmentation
2. In low-resource settings, simple probabilistic models with appropriate inductive biases can dramatically outperform data-hungry neural approaches
3. Treating segmentation as a language-specific phenomenon provides a powerful signal for language identification, especially for closely related varieties
4. Modular, incremental learning architectures offer practical advantages for production systems dealing with evolving language distributions
5. The gap between high-resource and low-resource LID performance can be substantially narrowed through sample-efficient methods, democratizing multilingual NLP capabilities