Technical Report: Adjudication of Coreference Annotations via Answer Set Optimization

TL;DR

This paper introduces an automatic method for merging multiple coreference annotations into a single gold standard using answer set programming, addressing linguistic constraints and optimization criteria.

Contribution

It presents the first automated approach for coreference annotation adjudication that incorporates linguistic constraints and optimization, demonstrated on Turkish datasets.

Findings

Successfully merged annotations on Turkish datasets.

Utilized answer set programming tools for optimal adjudication.

Provided benchmark datasets and evaluated the approach.

Abstract

We describe the first automatic approach for merging coreference annotations obtained from multiple annotators into a single gold standard. This merging is subject to certain linguistic hard constraints and optimization criteria that prefer solutions with minimal divergence from annotators. The representation involves an equivalence relation over a large number of elements. We use Answer Set Programming to describe two representations of the problem and four objective functions suitable for different datasets. We provide two structurally different real-world benchmark datasets based on the METU-Sabanci Turkish Treebank and we report our experiences in using the Gringo, Clasp, and Wasp tools for computing optimal adjudication results on these datasets.