Subjectivity in the Annotation of Bridging Anaphora

TL;DR

This paper investigates the inherent subjectivity in annotating bridging anaphora, highlighting annotation challenges, proposing a new classification system, and revealing under-annotation issues in existing resources.

Contribution

It introduces a new classification system for bridging subtypes and provides an analysis of annotation subjectivity and under-annotation in existing corpora.

Findings

Agreement on bridging subtype categories was moderate.

Annotator overlap for identifying bridging instances was low.

Many disagreements stemmed from subjective entity understanding.

Abstract

Bridging refers to the associative relationship between inferable entities in a discourse and the antecedents which allow us to understand them, such as understanding what "the door" means with respect to an aforementioned "house". As identifying associative relations between entities is an inherently subjective task, it is difficult to achieve consistent agreement in the annotation of bridging anaphora and their antecedents. In this paper, we explore the subjectivity involved in the annotation of bridging instances at three levels: anaphor recognition, antecedent resolution, and bridging subtype selection. To do this, we conduct an annotation pilot on the test set of the existing GUM corpus, and propose a newly developed classification system for bridging subtypes, which we compare to previously proposed schemes. Our results suggest that some previous resources are likely to be severely under-annotated. We also find that while agreement on the bridging subtype category was moderate, annotator overlap for exhaustively identifying instances of bridging is low, and that many disagreements resulted from subjective understanding of the entities involved.