On Crowdsourcing Task Design for Discourse Relation Annotation

TL;DR

This paper compares free-choice and forced-choice crowdsourcing methods for annotating implicit discourse relations, revealing that free-choice yields less diverse but more contextually flexible annotations, impacting discourse analysis.

Contribution

It provides a systematic comparison of two annotation strategies on a large corpus, highlighting how task design influences annotation diversity and interpretability.

Findings

Free-choice approach produces less diverse annotations.

Forced-choice approach increases annotation diversity.

Task design significantly affects annotator behavior and annotation quality.

Abstract

Interpreting implicit discourse relations involves complex reasoning, requiring the integration of semantic cues with background knowledge, as overt connectives like because or then are absent. These relations often allow multiple interpretations, best represented as distributions. In this study, we compare two established methods that crowdsource English implicit discourse relation annotation by connective insertion: a free-choice approach, which allows annotators to select any suitable connective, and a forced-choice approach, which asks them to select among a set of predefined options. Specifically, we re-annotate the whole DiscoGeM 1.0 corpus -- initially annotated with the free-choice method -- using the forced-choice approach. The free-choice approach allows for flexible and intuitive insertion of various connectives, which are context-dependent. Comparison among over 130,000 annotations, however, shows that the free-choice strategy produces less diverse annotations, often converging on common labels. Analysis of the results reveals the interplay between task design and the annotators' abilities to interpret and produce discourse relations.