EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Training

TL;DR

This paper demonstrates that continued pretraining and post-training alignment can significantly improve Estonian language capabilities in a multilingual LLM without sacrificing English performance.

Contribution

It introduces a method combining CPT and post-training alignment to enhance Estonian skills in a multilingual LLM while maintaining overall performance.

Findings

Significant improvements in Estonian linguistic competence and reasoning.

Enhanced translation quality and instruction-following in Estonian.

Maintained competitive performance on English benchmarks.

Abstract

Large language models (LLMs) are predominantly trained on English-centric data, resulting in uneven performance for smaller languages. We study whether continued pretraining (CPT) can substantially improve Estonian capabilities in a pretrained multilingual LLM while preserving its English and general reasoning performance. Using Llama 3.1 8B as the main base model, we perform CPT on a mixture that increases Estonian exposure while approximating the original training distribution through English replay and the inclusion of code, mathematics, and instruction-like data. We subsequently apply supervised fine-tuning, preference optimization, and chat vector merging to introduce robust instruction-following behavior. Evaluation on a comprehensive suite of Estonian benchmarks shows consistent gains in linguistic competence, knowledge, reasoning, translation quality, and instruction-following compared to the original base model and its instruction-tuned variant, while maintaining competitive performance on English benchmarks. These findings indicate that CPT, with an appropriately balanced data mixture, together with post-training alignment, can substantially improve single-language capabilities in pretrained multilingual LLMs.