A Large-Scale Corpus for Conversation Disentanglement

TL;DR

This paper introduces a large, manually annotated dataset of over 77,000 messages with reply-structure graphs to improve conversation disentanglement, addressing previous data limitations and revealing issues in existing dialogue corpora.

Contribution

The creation of the largest annotated dataset for conversation disentanglement, including adjudication and context, enabling more robust data-driven methods.

Findings

80% of conversations in a popular corpus are incomplete or contain extraneous messages

The dataset is 16 times larger than previous datasets

Includes adjudication of annotation disagreements and context information

Abstract

Disentangling conversations mixed together in a single stream of messages is a difficult task, made harder by the lack of large manually annotated datasets. We created a new dataset of 77,563 messages manually annotated with reply-structure graphs that both disentangle conversations and define internal conversation structure. Our dataset is 16 times larger than all previously released datasets combined, the first to include adjudication of annotation disagreements, and the first to include context. We use our data to re-examine prior work, in particular, finding that 80% of conversations in a widely used dialogue corpus are either missing messages or contain extra messages. Our manually-annotated data presents an opportunity to develop robust data-driven methods for conversation disentanglement, which will help advance dialogue research.