Action-Based Conversations Dataset: A Corpus for Building More In-Depth Task-Oriented Dialogue Systems

TL;DR

The paper introduces ABCD, a comprehensive dataset of over 10,000 customer service dialogues with detailed annotations, to advance the development of more realistic and effective task-oriented dialogue systems.

Contribution

It provides a new fully-labeled dataset with diverse user intents and proposes two novel dialogue tasks, establishing baselines with large-scale language models.

Findings

Sophisticated models outperform simpler ones.

A significant gap remains to reach human-level performance.

The dataset enables studying multi-step, policy-driven dialogues.

Abstract

Existing goal-oriented dialogue datasets focus mainly on identifying slots and values. However, customer support interactions in reality often involve agents following multi-step procedures derived from explicitly-defined company policies as well. To study customer service dialogue systems in more realistic settings, we introduce the Action-Based Conversations Dataset (ABCD), a fully-labeled dataset with over 10K human-to-human dialogues containing 55 distinct user intents requiring unique sequences of actions constrained by policies to achieve task success. We propose two additional dialog tasks, Action State Tracking and Cascading Dialogue Success, and establish a series of baselines involving large-scale, pre-trained language models on this dataset. Empirical results demonstrate that while more sophisticated networks outperform simpler models, a considerable gap (50.8% absolute accuracy) still exists to reach human-level performance on ABCD.