GigaChat Family: Efficient Russian Language Modeling Through Mixture of Experts Architecture

TL;DR

This paper introduces GigaChat, a family of efficient Russian language models based on Mixture of Experts architecture, with detailed architecture, training, and evaluation, including open-source models and system demonstrations.

Contribution

It presents the first large-scale Russian LLM family using Mixture of Experts, with detailed design, training, and open-source release to advance NLP research and applications.

Findings

GigaChat models outperform multilingual models on Russian benchmarks.

Open-source models enable broader research and industrial use.

System demos showcase practical deployment of GigaChat.

Abstract

Generative large language models (LLMs) have become crucial for modern NLP research and applications across various languages. However, the development of foundational models specifically tailored to the Russian language has been limited, primarily due to the significant computational resources required. This paper introduces the GigaChat family of Russian LLMs, available in various sizes, including base models and instruction-tuned versions. We provide a detailed report on the model architecture, pre-training process, and experiments to guide design choices. In addition, we evaluate their performance on Russian and English benchmarks and compare GigaChat with multilingual analogs. The paper presents a system demonstration of the top-performing models accessible via an API, a Telegram bot, and a Web interface. Furthermore, we have released three open GigaChat models in open-source (https://huggingface.co/ai-sage), aiming to expand NLP research opportunities and support the development of industrial solutions for the Russian language.