What GPT Knows About Who is Who

TL;DR

This paper investigates the ability of large language models like GPT-2 and GPT-Neo to perform coreference resolution using prompt engineering, revealing their limited and inconsistent capabilities in identifying coreferent mentions.

Contribution

It introduces a QA-based prompt-engineering approach to assess LLMs' coreference resolution abilities, highlighting their limitations and sensitivity to prompts.

Findings

GPT-2 and GPT-Neo can produce valid answers

Their coreference identification is limited and inconsistent

Performance is highly prompt-sensitive

Abstract

Coreference resolution -- which is a crucial task for understanding discourse and language at large -- has yet to witness widespread benefits from large language models (LLMs). Moreover, coreference resolution systems largely rely on supervised labels, which are highly expensive and difficult to annotate, thus making it ripe for prompt engineering. In this paper, we introduce a QA-based prompt-engineering method and discern \textit{generative}, pre-trained LLMs' abilities and limitations toward the task of coreference resolution. Our experiments show that GPT-2 and GPT-Neo can return valid answers, but that their capabilities to identify coreferent mentions are limited and prompt-sensitive, leading to inconsistent results.