Aligning Intraobserver Agreement by Transitivity

TL;DR

This paper introduces a transitivity-based method for measuring intraobserver agreement, offering a more bias-resistant alternative to traditional test-retest strategies in annotation tasks.

Contribution

It proposes a novel transitivity-based approach for assessing intraobserver agreement, including a representation theorem and applications in data collection design.

Findings

Transitivity measure is less sensitive to bias than test-retest methods.

Representation theorem maps transitive judgments to a measurement scale.

Application of transitivity reduces data collection complexity.

Abstract

Annotation reproducibility and accuracy rely on good consistency within annotators. We propose a novel method for measuring within annotator consistency or annotator Intraobserver Agreement (IA). The proposed approach is based on transitivity, a measure that has been thoroughly studied in the context of rational decision-making. The transitivity measure, in contrast with the commonly used test-retest strategy for annotator IA, is less sensitive to the several types of bias introduced by the test-retest strategy. We present a representation theorem to the effect that relative judgement data that meet transitivity can be mapped to a scale (in terms of measurement theory). We also discuss a further application of transitivity as part of data collection design for addressing the problem of the quadratic complexity of data collection of relative judgements.