Who Decides if AI is Fair? The Labels Problem in Algorithmic Auditing

TL;DR

This paper demonstrates that label quality significantly impacts the outcomes of AI algorithm audits, highlighting the need for standardized benchmarks to ensure fair and accurate evaluations in high-stakes settings.

Contribution

It provides empirical evidence on how label fidelity affects audit results and advocates for developing consensus-driven benchmarks for label quality.

Findings

Label quality can distort performance assessments in real-world audits.

Rigorous label cleaning can eliminate apparent disparities between groups.

Trade-offs between label quality and annotation costs influence audit reliability.

Abstract

Labelled "ground truth" datasets are routinely used to evaluate and audit AI algorithms applied in high-stakes settings. However, there do not exist widely accepted benchmarks for the quality of labels in these datasets. We provide empirical evidence that quality of labels can significantly distort the results of algorithmic audits in real-world settings. Using data annotators typically hired by AI firms in India, we show that fidelity of the ground truth data can lead to spurious differences in performance of ASRs between urban and rural populations. After a rigorous, albeit expensive, label cleaning process, these disparities between groups disappear. Our findings highlight how trade-offs between label quality and data annotation costs can complicate algorithmic audits in practice. They also emphasize the need for development of consensus-driven, widely accepted benchmarks for label quality.