Inter-Coder Agreement for Nominal Scales: A Model-based Approach

TL;DR

This paper introduces a model-based approach to inter-coder agreement for nominal scales, aiming to clarify reliability measures and address issues like bias and prevalence effects through a formal axiomatic framework.

Contribution

It proposes a formal model for inter-coder reliability, defines a new reliability measure as a probability, and provides an algorithm with simulations to evaluate its accuracy.

Findings

The model clarifies conditions for unique reliability determination.

The algorithm accurately estimates reliability under various settings.

Simulations demonstrate the effectiveness of the proposed approach.

Abstract

Inter-coder agreement measures, like Cohen's kappa, correct the relative frequency of agreement between coders to account for agreement which simply occurs by chance. However, in some situations these measures exhibit behavior which make their values difficult to interprete. These properties, e.g. the "annotator bias" or the "problem of prevalence", refer to a tendency of some of these measures to indicate counterintuitive high or low values of reliability depending on conditions which many researchers consider as unrelated to inter-coder reliability. However, not all researchers agree with this view, and since there is no commonly accepted formal definition of inter-coder reliability, it is hard to decide whether this depends upon a different concept of reliability or simply upon flaws in the measuring algorithms. In this note we therefore take an axiomatic approach: we introduce a model for the rating of items by several coders according to a nominal scale. Based upon this model we define inter-coder reliability as a probability to assign a category to an item with certainty. We then discuss under which conditions this notion of inter-coder reliability is uniquely determined given typical experimental results, i.e. relative frequencies of category assignments by different coders. In addition we provide an algorithm and conduct numerical simulations which exhibit the accuracy of this algorithm under different model parameter settings.