How Good is the Model in Model-in-the-loop Event Coreference Resolution Annotation?

TL;DR

This paper introduces a model-in-the-loop annotation approach for event coreference resolution, significantly reducing manual effort while maintaining high recall, validated through simulation and a novel evaluation metric.

Contribution

It proposes a new annotation method combining machine suggestions with human annotation, improving efficiency and recall in cross-document event coreference annotation.

Findings

Achieves 97% recall with reduced manual workload

Introduces a novel annotator-centric recall-annotation effort trade-off metric

Demonstrates effectiveness across various models and datasets

Abstract

Annotating cross-document event coreference links is a time-consuming and cognitively demanding task that can compromise annotation quality and efficiency. To address this, we propose a model-in-the-loop annotation approach for event coreference resolution, where a machine learning model suggests likely corefering event pairs only. We evaluate the effectiveness of this approach by first simulating the annotation process and then, using a novel annotator-centric Recall-Annotation effort trade-off metric, we compare the results of various underlying models and datasets. We finally present a method for obtaining 97\% recall while substantially reducing the workload required by a fully manual annotation process. Code and data can be found at https://github.com/ahmeshaf/model_in_coref