Estimating the Usefulness of Clarifying Questions and Answers for Conversational Search

TL;DR

This paper introduces a method to evaluate and incorporate user answers to clarifying questions in conversational search, improving retrieval performance and robustness against unhelpful responses.

Contribution

It proposes a classifier to assess answer usefulness and a transformer-based module for query rewriting, advancing mixed-initiative conversational search systems.

Findings

Significant performance improvements over non-mixed-initiative baselines

Effective filtering of unhelpful answers enhances retrieval accuracy

Robustness against non-useful responses is demonstrated

Abstract

While the body of research directed towards constructing and generating clarifying questions in mixed-initiative conversational search systems is vast, research aimed at processing and comprehending users' answers to such questions is scarce. To this end, we present a simple yet effective method for processing answers to clarifying questions, moving away from previous work that simply appends answers to the original query and thus potentially degrades retrieval performance. Specifically, we propose a classifier for assessing usefulness of the prompted clarifying question and an answer given by the user. Useful questions or answers are further appended to the conversation history and passed to a transformer-based query rewriting module. Results demonstrate significant improvements over strong non-mixed-initiative baselines. Furthermore, the proposed approach mitigates the performance drops when non useful questions and answers are utilized.