KaLM-Embedding-V2: Superior Training Techniques and Data Inspire A Versatile Embedding Model

TL;DR

KaLM-Embedding-V2 introduces a series of compact, versatile embedding models that leverage advanced training techniques and high-quality data to achieve state-of-the-art performance, rivaling much larger models.

Contribution

The paper presents novel training strategies and data curation methods that significantly enhance the performance of small-sized embedding models.

Findings

Achieves state-of-the-art results on the Massive Text Embedding Benchmark.

Outperforms comparable size models and rivals larger models by 3-26x.

Demonstrates the effectiveness of multi-stage training and high-quality data.

Abstract

Recent advancements in Large Language Models (LLMs)-based text embedding models primarily focus on data scaling or synthesis, yet limited exploration of training techniques and data quality, thereby constraining performance. In this work, we propose KaLM-Embedding-V2, a series of versatile and compact embedding models, systematically incentivizing advanced embedding capability in LLMs by superior training techniques and high-quality data. For model architecture, we implement the models on a 0.5B compact size with simple mean-pooling to produce fixed-length embeddings and remove the causal attention mask to enable fully bidirectional representation learning. For training techniques, we propose a progressive multi-stage training pipeline: pre-training on weakly supervised large-scale datasets, fine-tuning with supervised high-quality datasets, and contrastive distillation with fine-grained soft signals, integrated with focal-style reweighting and online hard-negative mixing to emphasize difficult samples and enrich hard negatives, respectively. For training data, we curate over 20 categories for pre-training and 100 categories for fine-tuning and contrastive distillation, to improve both performance and generalization, leveraging task-specific instructions, hard-negative mining, and example-based multi-class labeling to ensure high quality. Combining these techniques, our KaLM-Embedding-V2 series achieves state-of-the-art performance on the Massive Text Embedding Benchmark, outperforming models of comparable size and rivaling models 3-26x larger, setting a new standard for versatile and compact embedding models under 1B parameters.