NV-Retriever: Improving text embedding models with effective hard-negative mining

TL;DR

This paper introduces NV-Retriever, a novel hard-negative mining method for contrastive training of text embedding models, significantly improving retrieval accuracy and training efficiency in information retrieval tasks.

Contribution

It proposes positive-aware mining techniques that leverage relevance scores for better false negative removal, enhancing contrastive learning for text embeddings.

Findings

NV-Retriever-v1 scores 60.9 on MTEB Retrieval benchmark

Achieves state-of-the-art results on BEIR dataset

Faster training with improved retrieval accuracy

Abstract

Text embedding models have been popular for information retrieval applications such as semantic search and Question-Answering systems based on Retrieval-Augmented Generation (RAG). Those models are typically Transformer models that are fine-tuned with contrastive learning objectives. One of the challenging aspects of fine-tuning embedding models is the selection of high quality hard-negative passages for contrastive learning. In this paper we introduce a family of positive-aware mining methods that use the positive relevance score as an anchor for effective false negative removal, leading to faster training and more accurate retrieval models. We provide an ablation study on hard-negative mining methods over their configurations, exploring different teacher and base models. We further demonstrate the efficacy of our proposed mining methods at scale with the NV-Retriever-v1 model, which scores 60.9 on MTEB Retrieval (BEIR) benchmark and placed 1st when it was published to the MTEB Retrieval on July, 2024.