Toward LLMs Beyond English-Centric Development

TL;DR

This paper analyzes the language bias in large language models, showing that English dominance persists and that language-specific training may be more effective than continual pre-training for cultural understanding.

Contribution

It provides evidence that dedicated per-language training is more beneficial than continual pre-training for non-English LLMs, challenging common adaptation strategies.

Findings

LLMs are heavily biased toward English.

Continual pre-training does not reduce costs compared to training from scratch.

Per-language investment may be more effective for cultural understanding.

Abstract

Through an analysis of sequences generated by open-weight large language models (LLMs), we demonstrate that LLMs are heavily biased toward English. While continual pre-training is commonly used to adapt LLMs to a target language, we show that it does not offer a cost advantage over training from scratch, even for improving cultural understanding in the target language. These findings suggest that dedicated per-language investment may become increasingly important for future LLM development, rather than relying primarily on the expansion of English-centric resources.