Incorporating Annotator Uncertainty into Representations of Discourse Relations

TL;DR

This paper explores how incorporating annotator uncertainty and dialogue context into distributed representations improves modeling discourse relations in spoken conversations, addressing challenges faced by novice annotators.

Contribution

It introduces a method to embed annotator confidence and dialogue context into discourse relation representations, enhancing the understanding of annotation uncertainty.

Findings

Dialogue context significantly predicts confidence scores.

Weighted representations coherently model annotator uncertainty.

Hierarchical clustering reveals meaningful discourse relation groupings.

Abstract

Annotation of discourse relations is a known difficult task, especially for non-expert annotators. In this paper, we investigate novice annotators' uncertainty on the annotation of discourse relations on spoken conversational data. We find that dialogue context (single turn, pair of turns within speaker, and pair of turns across speakers) is a significant predictor of confidence scores. We compute distributed representations of discourse relations from co-occurrence statistics that incorporate information about confidence scores and dialogue context. We perform a hierarchical clustering analysis using these representations and show that weighting discourse relation representations with information about confidence and dialogue context coherently models our annotators' uncertainty about discourse relation labels.