Reliability Gaps Between Groups in COMPAS Dataset

TL;DR

This study examines how inter-rater reliability issues in risk assessment tools like COMPAS can lead to systematic differences in reliability between groups, influenced by the choice of statistical measure and correction methods.

Contribution

It introduces a simulation approach to assess the impact of inter-rater reliability on different groups within the COMPAS dataset, highlighting the dependence on statistical measures and correction techniques.

Findings

Systematic differences in reliability between groups were observed.

The sign of the reliability difference depends on the statistical measure used.

Correcting for group prevalence affects the observed reliability gaps.

Abstract

This paper investigates the inter-rater reliability of risk assessment instruments (RAIs). The main question is whether different, socially salient groups are affected differently by a lack of inter-rater reliability of RAIs, that is, whether mistakes with respect to different groups affects them differently. The question is investigated with a simulation study of the COMPAS dataset. A controlled degree of noise is injected into the input data of a predictive model; the noise can be interpreted as a synthetic rater that makes mistakes. The main finding is that there are systematic differences in output reliability between groups in the COMPAS dataset. The sign of the difference depends on the kind of inter-rater statistic that is used (Cohen's Kappa, Byrt's PABAK, ICC), and in particular whether or not a correction of predictions prevalences of the groups is used.