Analyzing and Characterizing User Intent in Information-seeking Conversations

TL;DR

This paper introduces MSDialog, a new dataset of over 2,000 annotated multi-turn QA dialogs from an online forum, to analyze user intent patterns in information-seeking conversations, aiding the development of conversational search systems.

Contribution

The paper presents a novel, publicly available dataset with detailed user intent annotations, enabling in-depth analysis of information-seeking conversation patterns.

Findings

Identification of recurring user intent patterns

Insights into intent co-occurrence and flow in dialogs

Potential applications for improving conversational search systems

Abstract

Understanding and characterizing how people interact in information-seeking conversations is crucial in developing conversational search systems. In this paper, we introduce a new dataset designed for this purpose and use it to analyze information-seeking conversations by user intent distribution, co-occurrence, and flow patterns. The MSDialog dataset is a labeled dialog dataset of question answering (QA) interactions between information seekers and providers from an online forum on Microsoft products. The dataset contains more than 2,000 multi-turn QA dialogs with 10,000 utterances that are annotated with user intent on the utterance level. Annotations were done using crowdsourcing. With MSDialog, we find some highly recurring patterns in user intent during an information-seeking process. They could be useful for designing conversational search systems. We will make our dataset freely available to encourage exploration of information-seeking conversation models.