On Event Individuation for Document-Level Information Extraction

TL;DR

This paper questions the suitability of template filling for document-level information extraction, highlighting challenges in event individuation that impact dataset quality, evaluation metrics, and model learning.

Contribution

It critically examines the assumptions behind template filling for document-level IE and discusses the implications of event individuation difficulties.

Findings

Human annotators disagree on event boundaries

Template filling metrics may not reflect true understanding

Datasets for document-level IE have quality issues

Abstract

As information extraction (IE) systems have grown more adept at processing whole documents, the classic task of template filling has seen renewed interest as benchmark for document-level IE. In this position paper, we call into question the suitability of template filling for this purpose. We argue that the task demands definitive answers to thorny questions of event individuation -- the problem of distinguishing distinct events -- about which even human experts disagree. Through an annotation study and error analysis, we show that this raises concerns about the usefulness of template filling metrics, the quality of datasets for the task, and the ability of models to learn it. Finally, we consider possible solutions.