GigaEmbeddings: Efficient Russian Language Embedding Model

TL;DR

GigaEmbeddings is a new high-performance Russian text embedding model that uses hierarchical instruction tuning, architectural innovations, and a three-stage training pipeline to achieve state-of-the-art results across multiple tasks.

Contribution

It introduces a novel framework with architectural and training innovations for efficient Russian language embeddings, unifying diverse objectives and leveraging synthetic data.

Findings

Achieves 69.1 average score on ruMTEB benchmark

Outperforms larger baseline models on multiple tasks

Maintains efficiency with 25% transformer layer pruning

Abstract

We introduce GigaEmbeddings, a novel framework for training high-performance Russian-focused text embeddings through hierarchical instruction tuning of the decoder-only LLM designed specifically for Russian language (GigaChat-3B). Our three-stage pipeline, comprising large-scale contrastive pre-training in web-scale corpora, fine-tuning with hard negatives, and multitask generalization across retrieval, classification, and clustering tasks, addresses key limitations of existing methods by unifying diverse objectives and leveraging synthetic data generation. Architectural innovations include bidirectional attention for contextual modeling, latent attention pooling for robust sequence aggregation, and strategic pruning of 25% of transformer layers to enhance efficiency without compromising performance. Evaluated on the ruMTEB benchmark spanning 23 multilingual tasks, GigaEmbeddings achieves state-of-the-art results (69.1 avg. score), outperforming strong baselines with a larger number of parameters.