ProCIS: A Benchmark for Proactive Retrieval in Conversations

TL;DR

This paper introduces ProCIS, a large-scale dataset and evaluation framework for proactive conversational information seeking, enabling systems to monitor conversations and proactively retrieve relevant resources at opportune moments.

Contribution

It presents a new dataset, ProCIS, with relevance annotations and a novel evaluation metric for proactive retrieval in multi-party conversations, addressing a gap in existing research.

Findings

ProCIS dataset contains over 2.8 million conversations.

Normalized proactive discounted cumulative gain (npDCG) effectively evaluates proactive retrieval.

Benchmark results demonstrate the effectiveness of various models, including a new proposed model.

Abstract

The field of conversational information seeking, which is rapidly gaining interest in both academia and industry, is changing how we interact with search engines through natural language interactions. Existing datasets and methods are mostly evaluating reactive conversational information seeking systems that solely provide response to every query from the user. We identify a gap in building and evaluating proactive conversational information seeking systems that can monitor a multi-party human conversation and proactively engage in the conversation at an opportune moment by retrieving useful resources and suggestions. In this paper, we introduce a large-scale dataset for proactive document retrieval that consists of over 2.8 million conversations. We conduct crowdsourcing experiments to obtain high-quality and relatively complete relevance judgments through depth-k pooling. We also collect annotations related to the parts of the conversation that are related to each document, enabling us to evaluate proactive retrieval systems. We introduce normalized proactive discounted cumulative gain (npDCG) for evaluating these systems, and further provide benchmark results for a wide range of models, including a novel model we developed for this task. We believe that the developed dataset, called ProCIS, paves the path towards developing proactive conversational information seeking systems.