Do the Findings of Document and Passage Retrieval Generalize to the Retrieval of Responses for Dialogues?

TL;DR

This study investigates whether established retrieval techniques for documents and passages are effective for dialogue response retrieval, considering the unique structure and query-response dynamics of conversational data.

Contribution

The paper provides a replicability analysis showing that most retrieval findings from document and passage tasks generalize to dialogue response retrieval, with some domain-specific considerations.

Findings

Document expansion outperforms no-expansion baseline

Dense retrieval underperforms sparse baselines in zero-shot settings

Hard negative sampling improves dense model training

Abstract

A number of learned sparse and dense retrieval approaches have recently been proposed and proven effective in tasks such as passage retrieval and document retrieval. In this paper we analyze with a replicability study if the lessons learned generalize to the retrieval of responses for dialogues, an important task for the increasingly popular field of conversational search. Unlike passage and document retrieval where documents are usually longer than queries, in response ranking for dialogues the queries (dialogue contexts) are often longer than the documents (responses). Additionally, dialogues have a particular structure, i.e. multiple utterances by different users. With these differences in mind, we here evaluate how generalizable the following major findings from previous works are: (F1) query expansion outperforms a no-expansion baseline; (F2) document expansion outperforms a no-expansion baseline; (F3) zero-shot dense retrieval underperforms sparse baselines; (F4) dense retrieval outperforms sparse baselines; (F5) hard negative sampling is better than random sampling for training dense models. Our experiments -- based on three different information-seeking dialogue datasets -- reveal that four out of five findings (F2-F5) generalize to our domain