A Taxonomy of Challenges to Curating Fair Datasets

TL;DR

This paper presents a detailed taxonomy of challenges faced during the curation of fair datasets in machine learning, based on interviews with curators, highlighting systemic issues and offering recommendations for improvement.

Contribution

It introduces a comprehensive taxonomy of practical challenges in fair dataset curation, derived from empirical interviews, which was previously underexplored.

Findings

Identifies key challenges in dataset curation for fairness

Highlights systemic issues affecting fair data practices

Provides recommendations for improving curation processes

Abstract

Despite extensive efforts to create fairer machine learning (ML) datasets, there remains a limited understanding of the practical aspects of dataset curation. Drawing from interviews with 30 ML dataset curators, we present a comprehensive taxonomy of the challenges and trade-offs encountered throughout the dataset curation lifecycle. Our findings underscore overarching issues within the broader fairness landscape that impact data curation. We conclude with recommendations aimed at fostering systemic changes to better facilitate fair dataset curation practices.