Gamayun's Path to Multilingual Mastery: Cost-Efficient Training of a 1.5B-Parameter LLM

TL;DR

Gamayun is a cost-efficient 1.5B-parameter multilingual language model trained from scratch on 2.5T tokens, achieving state-of-the-art results in Russian and competitive performance across multiple languages and tasks.

Contribution

The paper introduces a novel two-stage training strategy for small multilingual LLMs, improving cross-lingual performance with less training data and computational resources.

Findings

Outperforms larger models like LLaMA3.2-1B on benchmarks

Surpasses Qwen2.5-1.5B on multilingual and English tasks

Achieves state-of-the-art results in Russian, including MERA benchmark

Abstract

We present Gamayun, a 1.5B-parameter multilingual language model trained entirely from scratch on 2.5T tokens. Designed for efficiency and deployment in resource-constrained environments, Gamayun addresses the lack of research on small non-English-centric LLMs by adopting a novel two-stage pre-training strategy: balanced multilingual training for cross-lingual alignment, followed by high-quality English enrichment to transfer performance gains across languages. Our model supports 12 languages, with special focus on Russian. Despite a significantly smaller training budget than comparable models, Gamayun outperforms LLaMA3.2-1B (9T tokens) on all considered benchmarks, and surpasses Qwen2.5-1.5B (18T tokens) on a wide range of English and multilingual tasks. It matches or exceeds Qwen3 (36T tokens) on most tasks outside advanced STEM, achieving state-of-the-art results in Russian, including the MERA benchmark, among the models of comparable size (1-2B parameters).