Sparse and Dense Approaches for the Full-rank Retrieval of Responses for Dialogues

TL;DR

This paper compares sparse and dense retrieval methods for full-rank response retrieval in dialogues, emphasizing the importance of effective first-stage retrieval in large response sets and proposing techniques that improve retrieval performance.

Contribution

It introduces a comprehensive analysis of sparse and dense retrieval approaches for large-scale dialogue response retrieval, highlighting the effectiveness of learned response expansion and fine-tuned dense models.

Findings

Dense retrieval with intermediate training outperforms other methods.

Learned response expansion is a strong baseline for sparse retrieval.

Hard negatives sampling can negatively impact dense retrieval performance.

Abstract

Ranking responses for a given dialogue context is a popular benchmark in which the setup is to re-rank the ground-truth response over a limited set of $n$ responses, where $n$ is typically 10. The predominance of this setup in conversation response ranking has lead to a great deal of attention to building neural re-rankers, while the first-stage retrieval step has been overlooked. Since the correct answer is always available in the candidate list of $n$ responses, this artificial evaluation setup assumes that there is a first-stage retrieval step which is always able to rank the correct response in its top-$n$ list. In this paper we focus on the more realistic task of full-rank retrieval of responses, where $n$ can be up to millions of responses. We investigate both dialogue context and response expansion techniques for sparse retrieval, as well as zero-shot and fine-tuned dense retrieval approaches. Our findings based on three different information-seeking dialogue datasets reveal that a learned response expansion technique is a solid baseline for sparse retrieval. We find the best performing method overall to be dense retrieval with intermediate training, i.e. a step after the language model pre-training where sentence representations are learned, followed by fine-tuning on the target conversational data. We also investigate the intriguing phenomena that harder negatives sampling techniques lead to worse results for the fine-tuned dense retrieval models. The code and datasets are available at https://github.com/Guzpenha/transformer_rankers/tree/full_rank_retrieval_dialogues.