Investigating Data Interventions for Subgroup Fairness: An ICU Case Study

TL;DR

This study examines how combining data sources affects subgroup fairness in healthcare machine learning models, revealing that data addition can both improve and impair fairness and performance.

Contribution

It highlights the limitations of data scaling for fairness, compares data-centric and model-based strategies, and emphasizes the importance of combining both approaches.

Findings

Data addition can both help and hurt model fairness and performance.

Many intuitive data selection strategies are unreliable.

Combining data and model calibration improves subgroup performance.

Abstract

In high-stakes settings where machine learning models are used to automate decision-making about individuals, the presence of algorithmic bias can exacerbate systemic harm to certain subgroups of people. These biases often stem from the underlying training data. In practice, interventions to "fix the data" depend on the actual additional data sources available -- where many are less than ideal. In these cases, the effects of data scaling on subgroup performance become volatile, as the improvements from increased sample size are counteracted by the introduction of distribution shifts in the training set. In this paper, we investigate the limitations of combining data sources to improve subgroup performance within the context of healthcare. Clinical models are commonly trained on datasets comprised of patient electronic health record (EHR) data from different hospitals or admission departments. Across two such datasets, the eICU Collaborative Research Database and the MIMIC-IV dataset, we find that data addition can both help and hurt model fairness and performance, and many intuitive strategies for data selection are unreliable. We compare model-based post-hoc calibration and data-centric addition strategies to find that the combination of both is important to improve subgroup performance. Our work questions the traditional dogma of "better data" for overcoming fairness challenges by comparing and combining data- and model-based approaches.