Enhancing Retrieval Performance: An Ensemble Approach For Hard Negative Mining

TL;DR

This paper introduces a novel ensemble approach for hard negative mining in training cross-encoder models, significantly improving retrieval performance in domain-specific contexts and benefiting advanced LLM systems.

Contribution

It presents a robust hard negative mining technique tailored for enterprise datasets, enhancing the training efficiency and effectiveness of cross-encoder re-rank models.

Findings

Hard negative sampling outperforms random sampling in model training.

Learning both similarity and dissimilarity improves retrieval accuracy.

The approach benefits large language model systems like RAG and ReAct.

Abstract

Ranking consistently emerges as a primary focus in information retrieval research. Retrieval and ranking models serve as the foundation for numerous applications, including web search, open domain QA, enterprise domain QA, and text-based recommender systems. Typically, these models undergo training on triplets consisting of binary relevance assignments, comprising one positive and one negative passage. However, their utilization involves a context where a significantly more nuanced understanding of relevance is necessary, especially when re-ranking a large pool of potentially relevant passages. Although collecting positive examples through user feedback like impressions or clicks is straightforward, identifying suitable negative pairs from a vast pool of possibly millions or even billions of documents possess a greater challenge. Generating a substantial number of negative pairs is often necessary to maintain the high quality of the model. Several approaches have been suggested in literature to tackle the issue of selecting suitable negative pairs from an extensive corpus. This study focuses on explaining the crucial role of hard negatives in the training process of cross-encoder models, specifically aiming to explain the performance gains observed with hard negative sampling compared to random sampling. We have developed a robust hard negative mining technique for efficient training of cross-encoder re-rank models on an enterprise dataset which has domain specific context. We provide a novel perspective to enhance retrieval models, ultimately influencing the performance of advanced LLM systems like Retrieval-Augmented Generation (RAG) and Reasoning and Action Agents (ReAct). The proposed approach demonstrates that learning both similarity and dissimilarity simultaneously with cross-encoders improves performance of retrieval systems.