What Language is This? Ask Your Tokenizer

TL;DR

UniLID is a novel language identification method based on UnigramLM tokenization that is efficient, adaptable, and highly effective, especially in low-resource and dialect identification scenarios.

Contribution

It introduces UniLID, a new LID approach leveraging UnigramLM tokenization, supporting incremental learning and integration into existing pipelines, with superior performance in low-resource settings.

Findings

Achieves over 70% accuracy with only five samples per language

Outperforms baselines like fastText, GlotLID, and CLD3 on benchmarks

Significantly improves dialect identification accuracy

Abstract

Language Identification (LID) is an important component of many multilingual natural language processing pipelines, where it facilitates corpus curation, training data analysis, and cross-lingual evaluation of large language models. Despite near-perfect performance on high-resource languages, existing systems remain brittle in low-resource and closely related language settings. We introduce UniLID, a simple and efficient LID method based on the UnigramLM tokenization algorithm, leveraging its probabilistic framing, parameter estimation technique and inference strategy. In short, we learn language-conditional unigram distributions over a shared tokenizer vocabulary but treat segmentation as a language-specific phenomenon. Our formulation is data- and compute-efficient, supports incremental addition of new languages without retraining existing models, and can naturally be integrated into existing language model tokenization pipelines. Empirical evaluations against widely used baselines, including fastText, GlotLID, and CLD3, show that UniLID achieves competitive performance on standard benchmarks, substantially improves sample efficiency in low-resource settings - surpassing 70% accuracy with as few as five labeled samples per language - and delivers large gains on fine-grained dialect identification.