Towards Filling the Gap in Conversational Search: From Passage Retrieval to Conversational Response Generation

TL;DR

This paper develops a high-quality snippet-level annotation dataset for conversational search, enabling better training and evaluation of response generation models that synthesize relevant passages into concise answers.

Contribution

It introduces a refined annotation protocol and collects a large dataset of 1.8k question-paragraph pairs with high-quality annotations for conversational response generation.

Findings

Refined annotation protocol improves data quality

Collected 1.8k annotated question-paragraph pairs

Insights for future response-generation model design

Abstract

Research on conversational search has so far mostly focused on query rewriting and multi-stage passage retrieval. However, synthesizing the top retrieved passages into a complete, relevant, and concise response is still an open challenge. Having snippet-level annotations of relevant passages would enable both (1) the training of response generation models that are able to ground answers in actual statements and (2) the automatic evaluation of the generated responses in terms of completeness. In this paper, we address the problem of collecting high-quality snippet-level answer annotations for two of the TREC Conversational Assistance track datasets. To ensure quality, we first perform a preliminary annotation study, employing different task designs, crowdsourcing platforms, and workers with different qualifications. Based on the outcomes of this study, we refine our annotation protocol before proceeding with the full-scale data collection. Overall, we gather annotations for 1.8k question-paragraph pairs, each annotated by three independent crowd workers. The process of collecting data at this magnitude also led to multiple insights about the problem that can inform the design of future response-generation methods. This is an extended version of the article published with the same title in the Proceedings of CIKM'23.