Is this Dialogue Coherent? Learning from Dialogue Acts and Entities

TL;DR

This paper explores how humans perceive coherence in open-domain dialogues by creating a new annotated dataset and evaluating models that incorporate dialogue acts and entities to predict coherence.

Contribution

It introduces the SWBD-Coh corpus with coherence annotations and demonstrates that combining dialogue acts and entity information improves coherence prediction models.

Findings

Models with both DA and entity info perform best.

Entity and DA patterns influence coherence perception.

SWBD-Coh dataset enables detailed coherence analysis.

Abstract

In this work, we investigate the human perception of coherence in open-domain dialogues. In particular, we address the problem of annotating and modeling the coherence of next-turn candidates while considering the entire history of the dialogue. First, we create the Switchboard Coherence (SWBD-Coh) corpus, a dataset of human-human spoken dialogues annotated with turn coherence ratings, where next-turn candidate utterances ratings are provided considering the full dialogue context. Our statistical analysis of the corpus indicates how turn coherence perception is affected by patterns of distribution of entities previously introduced and the Dialogue Acts used. Second, we experiment with different architectures to model entities, Dialogue Acts and their combination and evaluate their performance in predicting human coherence ratings on SWBD-Coh. We find that models combining both DA and entity information yield the best performances both for response selection and turn coherence rating.