WikiCREM: A Large Unsupervised Corpus for Coreference Resolution

TL;DR

WikiCREM is a large, automatically generated dataset for pronoun coreference resolution that enables training and evaluation of models, leading to state-of-the-art results on multiple benchmarks.

Contribution

The paper introduces WikiCREM, a novel large-scale unsupervised dataset for pronoun resolution, and demonstrates its effectiveness with models that outperform previous approaches.

Findings

Achieved state-of-the-art results on 6 out of 7 coreference datasets.

Demonstrated the effectiveness of the WikiCREM dataset for training coreference models.

Provided an off-the-shelf model for pronoun disambiguation.

Abstract

Pronoun resolution is a major area of natural language understanding. However, large-scale training sets are still scarce, since manually labelling data is costly. In this work, we introduce WikiCREM (Wikipedia CoREferences Masked) a large-scale, yet accurate dataset of pronoun disambiguation instances. We use a language-model-based approach for pronoun resolution in combination with our WikiCREM dataset. We compare a series of models on a collection of diverse and challenging coreference resolution problems, where we match or outperform previous state-of-the-art approaches on 6 out of 7 datasets, such as GAP, DPR, WNLI, PDP, WinoBias, and WinoGender. We release our model to be used off-the-shelf for solving pronoun disambiguation.