EasyECR: A Library for Easy Implementation and Evaluation of Event Coreference Resolution Models

TL;DR

EasyECR is an open-source library that standardizes event coreference resolution pipelines and datasets, enabling fair evaluation and promoting the development of more robust models across diverse domains.

Contribution

This work introduces EasyECR, the first library to unify ECR pipelines and datasets, facilitating easier implementation, evaluation, and comparison of models in the field.

Findings

ECR pipelines do not generalize well across datasets.

Model choice significantly impacts pipeline performance.

Reproducing ECR results is challenging without standardized tools.

Abstract

Event Coreference Resolution (ECR) is the task of clustering event mentions that refer to the same real-world event. Despite significant advancements, ECR research faces two main challenges: limited generalizability across domains due to narrow dataset evaluations, and difficulties in comparing models within diverse ECR pipelines. To address these issues, we develop EasyECR, the first open-source library designed to standardize data structures and abstract ECR pipelines for easy implementation and fair evaluation. More specifically, EasyECR integrates seven representative pipelines and ten popular benchmark datasets, enabling model evaluations on various datasets and promoting the development of robust ECR pipelines. By conducting extensive evaluation via our EasyECR, we find that, \lowercase\expandafter{\romannumeral1}) the representative ECR pipelines cannot generalize across multiple datasets, hence evaluating ECR pipelines on multiple datasets is necessary, \lowercase\expandafter{\romannumeral2}) all models in ECR pipelines have a great effect on pipeline performance, therefore, when one model in ECR pipelines are compared, it is essential to ensure that the other models remain consistent. Additionally, reproducing ECR results is not trivial, and the developed library can help reduce this discrepancy. The experimental results provide valuable baselines for future research.