Sparse Probability of Agreement

TL;DR

The paper introduces Sparse Probability of Agreement (SPA), a new metric for estimating inter-annotator agreement in datasets with incomplete annotations, ensuring unbiased estimates under certain conditions.

Contribution

It proposes SPA, a novel agreement metric that handles sparse annotation data and provides unbiased estimates with multiple weighing schemes.

Findings

SPA is an unbiased estimator under certain conditions.

Multiple weighing schemes improve SPA's flexibility.

SPA effectively estimates agreement in incomplete datasets.

Abstract

Measuring inter-annotator agreement is important for annotation tasks, but many metrics require a fully-annotated set of data, where all annotators annotate all samples. We define Sparse Probability of Agreement, SPA, which estimates the probability of agreement when not all annotator-item-pairs are available. We show that under certain conditions, SPA is an unbiased estimator, and we provide multiple weighing schemes for handling data with various degrees of annotation.