Evaluating Mixed-initiative Conversational Search Systems via User Simulation

TL;DR

This paper introduces USi, a user simulator for automatically evaluating conversational search systems by generating user responses to clarifying questions, reducing reliance on human evaluation and enabling multi-turn interaction studies.

Contribution

The paper presents USi, a GPT2-based user simulator for automatic evaluation of conversational search systems, and extends datasets for multi-turn interactions with crowdsourced data.

Findings

USi responses align well with user needs and match human answers.

USi performs accurately in single-turn clarifying question scenarios.

The approach facilitates multi-turn interaction research in conversational search.

Abstract

Clarifying the underlying user information need by asking clarifying questions is an important feature of modern conversational search system. However, evaluation of such systems through answering prompted clarifying questions requires significant human effort, which can be time-consuming and expensive. In this paper, we propose a conversational User Simulator, called USi, for automatic evaluation of such conversational search systems. Given a description of an information need, USi is capable of automatically answering clarifying questions about the topic throughout the search session. Through a set of experiments, including automated natural language generation metrics and crowdsourcing studies, we show that responses generated by USi are both inline with the underlying information need and comparable to human-generated answers. Moreover, we make the first steps towards multi-turn interactions, where conversational search systems asks multiple questions to the (simulated) user with a goal of clarifying the user need. To this end, we expand on currently available datasets for studying clarifying questions, i.e., Qulac and ClariQ, by performing a crowdsourcing-based multi-turn data acquisition. We show that our generative, GPT2-based model, is capable of providing accurate and natural answers to unseen clarifying questions in the single-turn setting and discuss capabilities of our model in the multi-turn setting. We provide the code, data, and the pre-trained model to be used for further research on the topic.