Training Dense Retrievers with Multiple Positive Passages

TL;DR

This paper systematically studies multi-positive training objectives for dense retrievers, demonstrating that the LSEPair loss offers superior robustness and performance by effectively leveraging dense supervision signals from LLMs and human annotations.

Contribution

It unifies and analyzes multiple multi-positive training objectives within a contrastive learning framework, providing theoretical insights and practical guidelines for improving retriever training with dense supervision.

Findings

LSEPair consistently outperforms other objectives in robustness and accuracy.

JointLH and SumMargLH are sensitive to positive quality, affecting performance.

Random sampling (Rand1LH) is a reliable baseline for training.

Abstract

Modern knowledge-intensive systems, such as retrieval-augmented generation (RAG), rely on effective retrievers to establish the performance ceiling for downstream modules. However, retriever training has been bottlenecked by sparse, single-positive annotations, which lead to false-negative noise and suboptimal supervision. While the advent of large language models (LLMs) makes it feasible to collect comprehensive multi-positive relevance labels at scale, the optimal strategy for incorporating these dense signals into training remains poorly understood. In this paper, we present a systematic study of multi-positive optimization objectives for retriever training. We unify representative objectives, including Joint Likelihood (JointLH), Summed Marginal Likelihood (SumMargLH), and Log-Sum-Exp Pairwise (LSEPair) loss, under a shared contrastive learning framework. Our theoretical analysis characterizes their distinct gradient behaviors, revealing how each allocates probability mass across positive document sets. Empirically, we conduct extensive evaluations on Natural Questions, MS MARCO, and the BEIR benchmark across two realistic regimes: homogeneous LLM-annotated data and heterogeneous mixtures of human and LLM labels. Our results show that LSEPair consistently achieves superior robustness and performance across settings, while JointLH and SumMargLH exhibit high sensitivity to the quality of positives. Furthermore, we find that the simple strategy of random sampling (Rand1LH) serves as a reliable baseline. By aligning theoretical insights with empirical findings, we provide practical design principles for leveraging dense, LLM-augmented supervision to enhance retriever effectiveness.