Solving the Challenge Set without Solving the Task: On Winograd Schemas as a Test of Pronominal Coreference Resolution

TL;DR

This paper challenges the assumption that high performance on Winograd Schemas indicates overall pronominal coreference resolution ability, showing that models perform differently across datasets and proposing an ensemble approach for better accuracy.

Contribution

It reveals the limitations of current prompted language models on coreference tasks and introduces an ensemble method combining LM prompts with supervised systems for improved performance.

Findings

Prompted LMs perform poorly on certain coreference ambiguities.

Ensembling LM prompts with supervised models improves accuracy.

Dataset-specific evaluations are necessary for comprehensive system assessment.

Abstract

Challenge sets such as the Winograd Schema Challenge (WSC) are used to benchmark systems' ability to resolve ambiguities in natural language. If one assumes as in existing work that solving a given challenge set is at least as difficult as solving some more general task, then high performance on the challenge set should indicate high performance on the general task overall. However, we show empirically that this assumption of difficulty does not always hold. In particular, we demonstrate that despite the strong performance of prompted language models (LMs) on the WSC and its variants, these same modeling techniques perform relatively poorly at resolving certain pronominal ambiguities attested in OntoNotes and related datasets that are perceived to be easier. Motivated by these findings, we propose a method for ensembling a prompted LM with a supervised, task-specific system that is overall more accurate at resolving pronominal coreference across datasets. Finally, we emphasize that datasets involving the same linguistic phenomenon draw on distinct, but overlapping, capabilities, and evaluating on any one dataset alone does not provide a complete picture of a system's overall capability.