Lazy Data Practices Harm Fairness Research

TL;DR

This paper critically examines how common, unreflective data practices in fair ML hinder research reliability and fairness, highlighting issues of representation, minority exclusion, and opaque data handling, and proposes recommendations for responsible data use.

Contribution

It provides a systematic analysis of dataset usage in fair ML, identifying key shortcomings and offering guidelines to improve transparency and inclusivity in data practices.

Findings

Protected attribute representation is often lacking.

Minorities are frequently excluded during preprocessing.

Opaque data handling threatens fairness research generalization.

Abstract

Data practices shape research and practice on fairness in machine learning (fair ML). Critical data studies offer important reflections and critiques for the responsible advancement of the field by highlighting shortcomings and proposing recommendations for improvement. In this work, we present a comprehensive analysis of fair ML datasets, demonstrating how unreflective yet common practices hinder the reach and reliability of algorithmic fairness findings. We systematically study protected information encoded in tabular datasets and their usage in 280 experiments across 142 publications. Our analyses identify three main areas of concern: (1) a \textbf{lack of representation for certain protected attributes} in both data and evaluations; (2) the widespread \textbf{exclusion of minorities} during data preprocessing; and (3) \textbf{opaque data processing} threatening the generalization of fairness research. By conducting exemplary analyses on the utilization of prominent datasets, we demonstrate how unreflective data decisions disproportionately affect minority groups, fairness metrics, and resultant model comparisons. Additionally, we identify supplementary factors such as limitations in publicly available data, privacy considerations, and a general lack of awareness, which exacerbate these challenges. To address these issues, we propose a set of recommendations for data usage in fairness research centered on transparency and responsible inclusion. This study underscores the need for a critical reevaluation of data practices in fair ML and offers directions to improve both the sourcing and usage of datasets.