A Decomposition-Based Approach for Evaluating and Analyzing Inter-Annotator Disagreement

TL;DR

This paper introduces a new decomposition-based method to analyze inter-annotator disagreement at different levels, aiding understanding of annotation challenges and revealing latent structures within datasets.

Contribution

The paper presents a novel approach with two strategies for decomposing annotations, enabling detailed analysis of disagreement sources and uncovering underlying annotation structures.

Findings

Effective decomposition reveals sources of disagreement.

Two strategies provide flexible analysis options.

Application to narrative dataset demonstrates utility.

Abstract

We propose a novel method to conceptually decompose an existing annotation into separate levels, allowing the analysis of inter-annotators disagreement in each level separately. We suggest two distinct strategies in order to actualize this approach: a theoretically-driven one, in which the researcher defines a decomposition based on prior knowledge of the annotation task, and an exploration-based one, in which many possible decompositions are inductively computed and presented to the researcher for interpretation and evaluation. Utilizing a recently constructed dataset for narrative analysis as our use-case, we apply each of the two strategies to demonstrate the potential of our approach in testing hypotheses regarding the sources of annotation disagreements, as well as revealing latent structures and relations within the annotation task. We conclude by suggesting how to extend and generalize our approach, as well as use it for other purposes.