An Empirical Study of Clarifying Question-Based Systems

TL;DR

This empirical study investigates user willingness and behavior in answering clarifying questions in search and recommender systems, revealing insights into user engagement, fatigue, and answer accuracy to inform future system design.

Contribution

The paper provides the first empirical analysis of user responses to clarifying questions in interactive systems, highlighting practical user behavior patterns and challenges.

Findings

Users answer 11-21 questions on average.

Most users answer until reaching the target product.

12-17% of answers are opposite to the target description.

Abstract

Search and recommender systems that take the initiative to ask clarifying questions to better understand users' information needs are receiving increasing attention from the research community. However, to the best of our knowledge, there is no empirical study to quantify whether and to what extent users are willing or able to answer these questions. In this work, we conduct an online experiment by deploying an experimental system, which interacts with users by asking clarifying questions against a product repository. We collect both implicit interaction behavior data and explicit feedback from users showing that: (a) users are willing to answer a good number of clarifying questions (11-21 on average), but not many more than that; (b) most users answer questions until they reach the target product, but also a fraction of them stops due to fatigue or due to receiving irrelevant questions; (c) part of the users' answers (12-17%) are actually opposite to the description of the target product; while (d) most of the users (66-84%) find the question-based system helpful towards completing their tasks. Some of the findings of the study contradict current assumptions on simulated evaluations in the field, while they point towards improvements in the evaluation framework and can inspire future interactive search/recommender system designs.