Improving Document Retrieval Coherence for Semantically Equivalent Queries

TL;DR

This paper introduces a modified training loss for dense retrieval models that enhances their consistency in retrieving the same documents for semantically equivalent queries, leading to more stable and accurate retrieval.

Contribution

It proposes a variation of the Multi-Negative Ranking loss that improves retrieval coherence for semantically similar queries, reducing sensitivity to query variations.

Findings

Models trained with the new loss show lower sensitivity to query variations.

The new loss leads to higher retrieval accuracy across multiple datasets.

Enhanced coherence improves robustness of document retrieval systems.

Abstract

Dense Retrieval (DR) models have proven to be effective for Document Retrieval and Information Grounding tasks. Usually, these models are trained and optimized for improving the relevance of top-ranked documents for a given query. Previous work has shown that popular DR models are sensitive to the query and document lexicon: small variations of it may lead to a significant difference in the set of retrieved documents. In this paper, we propose a variation of the Multi-Negative Ranking loss for training DR that improves the coherence of models in retrieving the same documents with respect to semantically similar queries. The loss penalizes discrepancies between the top-k ranked documents retrieved for diverse but semantic equivalent queries. We conducted extensive experiments on various datasets, MS-MARCO, Natural Questions, BEIR, and TREC DL 19/20. The results show that (i) models optimizes by our loss are subject to lower sensitivity, and, (ii) interestingly, higher accuracy.