The Validity of Coreference-based Evaluations of Natural Language Understanding

TL;DR

This paper critically examines coreference-based evaluations of NLP systems, revealing their limitations in measurement validity and proposing a new evaluation method to better assess models' understanding of event plausibility.

Contribution

It identifies issues with existing coreference evaluations and introduces a novel evaluation focusing on event plausibility inference to improve assessment accuracy.

Findings

Standard benchmarks show models perform well but lack generalization.

Models are sensitive to evaluation conditions and often fail to generalize.

Current evaluation methods have validity issues, limiting conclusions.

Abstract

In this thesis, I refine our understanding as to what conclusions we can reach from coreference-based evaluations by expanding existing evaluation practices and considering the extent to which evaluation results are either converging or conflicting. First, I analyze standard coreference evaluations and show that their design often leads to non-generalizable conclusions due to issues of measurement validity - including contestedness (multiple, competing definitions of coreference) and convergent validity (evaluation results that rank models differently across benchmarks). Second, I propose and implement a novel evaluation focused on testing systems' ability to infer the relative plausibility of events, a key aspect of resolving coreference. Through this extended evaluation, I find that contemporary language models demonstrate strong performance on standard benchmarks - improving over earlier baseline systems within certain domains and types of coreference - but remain sensitive to the evaluation conditions: they often fail to generalize in ways one would expect a human to be capable of when evaluation contexts are slightly modified. Taken together, these findings clarify both the strengths, such as improved accuracy over baselines on widely used evaluations, and the limitations of the current NLP paradigm, including weaknesses in measurement validity, and suggest directions for future work in developing better evaluation methods and more genuinely generalizable systems.