Model-based annotation of coreference

TL;DR

This paper introduces a model-based approach to coreference annotation that links entities to a knowledge base, simplifying the task, increasing efficiency and agreement, and providing new benchmark datasets for evaluation.

Contribution

It proposes a novel model-based annotation method for coreference, especially pronouns, and provides new datasets and evaluation of state-of-the-art resolvers.

Findings

Model-based annotation speeds up the annotation process.

It results in higher inter-annotator agreement.

New benchmark datasets for coreference resolution are introduced.

Abstract

Humans do not make inferences over texts, but over models of what texts are about. When annotators are asked to annotate coreferent spans of text, it is therefore a somewhat unnatural task. This paper presents an alternative in which we preprocess documents, linking entities to a knowledge base, and turn the coreference annotation task -- in our case limited to pronouns -- into an annotation task where annotators are asked to assign pronouns to entities. Model-based annotation is shown to lead to faster annotation and higher inter-annotator agreement, and we argue that it also opens up for an alternative approach to coreference resolution. We present two new coreference benchmark datasets, for English Wikipedia and English teacher-student dialogues, and evaluate state-of-the-art coreference resolvers on them.