Cross-replication Reliability -- An Empirical Approach to Interpreting Inter-rater Reliability

TL;DR

This paper introduces the xRR framework, an empirical method for interpreting inter-rater reliability by benchmarking against baseline measures, demonstrated on a large facial expression dataset.

Contribution

It proposes a novel cross-replication reliability (xRR) measure based on Cohen's kappa and provides an open dataset for evaluating IRR in crowdsourced annotations.

Findings

xRR provides a new way to interpret IRR in context

Benchmarking against baseline measures improves IRR assessment

Open dataset enables further research in crowdsourced data quality

Abstract

We present a new approach to interpreting IRR that is empirical and contextualized. It is based upon benchmarking IRR against baseline measures in a replication, one of which is a novel cross-replication reliability (xRR) measure based on Cohen's kappa. We call this approach the xRR framework. We opensource a replication dataset of 4 million human judgements of facial expressions and analyze it with the proposed framework. We argue this framework can be used to measure the quality of crowdsourced datasets.