A Corpus-Based Investigation of Definite Description Use

TL;DR

This study investigates how definite descriptions are used in written texts, assessing annotation feasibility and agreement levels, revealing many discourse-new definites and questioning evaluation strategies for interpretation systems.

Contribution

It provides empirical data on annotating definite descriptions, compares classification schemes, and highlights challenges in agreement and interpretation in corpus annotation.

Findings

Low inter-annotator agreement with detailed schemes (K=0.63)

Higher agreement with simplified scheme (K=0.76)

High prevalence of discourse-new definites (around 50%)

Abstract

We present the results of a study of definite descriptions use in written texts aimed at assessing the feasibility of annotating corpora with information about definite description interpretation. We ran two experiments, in which subjects were asked to classify the uses of definite descriptions in a corpus of 33 newspaper articles, containing a total of 1412 definite descriptions. We measured the agreement among annotators about the classes assigned to definite descriptions, as well as the agreement about the antecedent assigned to those definites that the annotators classified as being related to an antecedent in the text. The most interesting result of this study from a corpus annotation perspective was the rather low agreement (K=0.63) that we obtained using versions of Hawkins' and Prince's classification schemes; better results (K=0.76) were obtained using the simplified scheme proposed by Fraurud that includes only two classes, first-mention and subsequent-mention. The agreement about antecedents was also not complete. These findings raise questions concerning the strategy of evaluating systems for definite description interpretation by comparing their results with a standardized annotation. From a linguistic point of view, the most interesting observations were the great number of discourse-new definites in our corpus (in one of our experiments, about 50% of the definites in the collection were classified as discourse-new, 30% as anaphoric, and 18% as associative/bridging) and the presence of definites which did not seem to require a complete disambiguation.