Bridging the Training-Inference Gap for Dense Phrase Retrieval

TL;DR

This paper addresses the mismatch between training objectives and inference scenarios in dense phrase retrieval, proposing an efficient validation method that improves retrieval accuracy for open-domain question answering.

Contribution

It introduces a practical validation approach using a small subset of data, reducing the training-inference gap and enhancing retrieval performance.

Findings

Top-1 phrase retrieval accuracy improved by 2-3 points.

Top-20 passage retrieval accuracy improved by 2-4 points.

Efficient validation enables better training strategies for dense retrievers.

Abstract

Building dense retrievers requires a series of standard procedures, including training and validating neural models and creating indexes for efficient search. However, these procedures are often misaligned in that training objectives do not exactly reflect the retrieval scenario at inference time. In this paper, we explore how the gap between training and inference in dense retrieval can be reduced, focusing on dense phrase retrieval (Lee et al., 2021) where billions of representations are indexed at inference. Since validating every dense retriever with a large-scale index is practically infeasible, we propose an efficient way of validating dense retrievers using a small subset of the entire corpus. This allows us to validate various training strategies including unifying contrastive loss terms and using hard negatives for phrase retrieval, which largely reduces the training-inference discrepancy. As a result, we improve top-1 phrase retrieval accuracy by 2~3 points and top-20 passage retrieval accuracy by 2~4 points for open-domain question answering. Our work urges modeling dense retrievers with careful consideration of training and inference via efficient validation while advancing phrase retrieval as a general solution for dense retrieval.