Vikhr: The Family of Open-Source Instruction-Tuned Large Language Models for Russian

TL;DR

Vikhr introduces a family of open-source bilingual LLMs optimized for Russian, utilizing adapted tokenizers and comprehensive training to improve performance and efficiency over previous models.

Contribution

The paper presents a novel pipeline for adapting English-oriented models to Russian, resulting in Vikhr, which outperforms prior models by training all weights and expanding instruction datasets.

Findings

Enhanced performance on Russian language tasks.

Significant improvements in computational efficiency.

Open-source availability of models and datasets.

Abstract

There has been a surge in the development of various Large Language Models (LLMs). However, text generation for languages other than English often faces significant challenges, including poor generation quality and reduced computational performance due to the disproportionate representation of tokens in the model's vocabulary. In this work, we address these issues by developing a pipeline for the adaptation of English-oriented pre-trained models to other languages and constructing efficient bilingual LLMs. Using this pipeline, we construct Vikhr, a series of bilingual open-source instruction-following LLMs designed specifically for the Russian language. ``Vikhr'' refers to the name of the Mistral LLM series and means a ``strong gust of wind.'' Unlike previous Russian-language models that typically rely on LoRA adapters on top of English-oriented models, sacrificing performance for lower training costs, Vikhr features an adapted tokenizer vocabulary and undergoes the continued pre-training and instruction tuning of all weights. This not only enhances the model's performance but also significantly improves its computational and contextual efficiency. We also expanded the instruction datasets and corpora for continued pre-training. The model weights, instruction sets, and code are publicly available.