Exploring the interplay of label bias with subgroup size and separability: A case study in mammographic density classification

TL;DR

This study examines how label bias in medical imaging datasets affects deep learning models, revealing that subgroup size and separability significantly influence learned features and fairness outcomes.

Contribution

It provides the first detailed analysis of how label bias impacts feature representations and subgroup performance in medical imaging AI.

Findings

Label bias causes significant shifts in learned feature representations.

Subgroup size and separability affect the extent of bias impact.

Validation set label quality influences subgroup performance metrics.

Abstract

Systematic mislabelling affecting specific subgroups (i.e., label bias) in medical imaging datasets represents an understudied issue concerning the fairness of medical AI systems. In this work, we investigated how size and separability of subgroups affected by label bias influence the learned features and performance of a deep learning model. Therefore, we trained deep learning models for binary tissue density classification using the EMory BrEast imaging Dataset (EMBED), where label bias affected separable subgroups (based on imaging manufacturer) or non-separable "pseudo-subgroups". We found that simulated subgroup label bias led to prominent shifts in the learned feature representations of the models. Importantly, these shifts within the feature space were dependent on both the relative size and the separability of the subgroup affected by label bias. We also observed notable differences in subgroup performance depending on whether a validation set with clean labels was used to define the classification threshold for the model. For instance, with label bias affecting the majority separable subgroup, the true positive rate for that subgroup fell from 0.898, when the validation set had clean labels, to 0.518, when the validation set had biased labels. Our work represents a key contribution toward understanding the consequences of label bias on subgroup fairness in medical imaging AI.