A Survey on Asking Clarification Questions Datasets in Conversational Systems

TL;DR

This survey reviews existing datasets, evaluation metrics, and benchmarks for Asking Clarification Questions in conversational systems, highlighting inconsistencies and proposing directions for future research to improve understanding of user intent.

Contribution

It provides a comprehensive comparison of ACQs datasets, evaluation strategies, and benchmarks, addressing current research limitations and guiding future developments in conversational systems.

Findings

Identified inconsistencies in datasets and evaluation methods.

Compared multiple ACQs-related benchmarks and tasks.

Discussed future research directions for ACQs in conversational AI.

Abstract

The ability to understand a user's underlying needs is critical for conversational systems, especially with limited input from users in a conversation. Thus, in such a domain, Asking Clarification Questions (ACQs) to reveal users' true intent from their queries or utterances arise as an essential task. However, it is noticeable that a key limitation of the existing ACQs studies is their incomparability, from inconsistent use of data, distinct experimental setups and evaluation strategies. Therefore, in this paper, to assist the development of ACQs techniques, we comprehensively analyse the current ACQs research status, which offers a detailed comparison of publicly available datasets, and discusses the applied evaluation metrics, joined with benchmarks for multiple ACQs-related tasks. In particular, given a thorough analysis of the ACQs task, we discuss a number of corresponding research directions for the investigation of ACQs as well as the development of conversational systems.