Sailor2: Sailing in South-East Asia with Inclusive Multilingual LLMs

TL;DR

Sailor2 is a family of multilingual language models tailored for South-East Asian languages, achieving competitive performance and providing a comprehensive guide for efficient development of inclusive LLMs.

Contribution

It introduces Sailor2 models with extensive SEA language support and offers a detailed cookbook for developing inclusive multilingual LLMs.

Findings

Sailor2-20B outperforms GPT-4o on SEA languages.

Models support 13 SEA languages while maintaining Chinese and English proficiency.

Provides a practical methodology for building inclusive multilingual LLMs.

Abstract

Sailor2 is a family of cutting-edge multilingual language models for South-East Asian (SEA) languages, available in 1B, 8B, and 20B sizes to suit diverse applications. Building on Qwen2.5, Sailor2 undergoes continuous pre-training on 500B tokens (400B SEA-specific and 100B replay tokens) to support 13 SEA languages while retaining proficiency in Chinese and English. Sailor2-20B model achieves a 50-50 win rate against GPT-4o across SEA languages. We also deliver a comprehensive cookbook on how to develop the multilingual model in an efficient manner, including five key aspects: data curation, pre-training, post-training, model customization and evaluation. We hope that Sailor2 model (Apache 2.0 license) will drive language development in the SEA region, and Sailor2 cookbook will inspire researchers to build more inclusive LLMs for other under-served languages.