Investigating Label Bias and Representational Sources of Age-Related Disparities in Medical Segmentation

TL;DR

This paper investigates the sources of age-related disparities in medical image segmentation, revealing systemic bias learned from biased labels and emphasizing the need to address qualitative differences for fairness.

Contribution

It introduces a framework for diagnosing bias in medical segmentation and demonstrates that bias stems from qualitative distributional differences, not just label quality or case difficulty.

Findings

Bias is learned and amplified from biased labels.

Balancing data by difficulty does not reduce age disparity.

Systemic bias originates from qualitative differences in patient cases.

Abstract

Algorithmic bias in medical imaging can perpetuate health disparities, yet its causes remain poorly understood in segmentation tasks. While fairness has been extensively studied in classification, segmentation remains underexplored despite its clinical importance. In breast cancer segmentation, models exhibit significant performance disparities against younger patients, commonly attributed to physiological differences in breast density. We audit the MAMA-MIA dataset, establishing a quantitative baseline of age-related bias in its automated labels, and reveal a critical Biased Ruler effect where systematically flawed labels for validation misrepresent a model's actual bias. However, whether this bias originates from lower-quality annotations (label bias) or from fundamentally more challenging image characteristics remains unclear. Through controlled experiments, we systematically refute hypotheses that the bias stems from label quality sensitivity or quantitative case difficulty imbalance. Balancing training data by difficulty fails to mitigate the disparity, revealing that younger patient cases are intrinsically harder to learn. We provide direct evidence that systemic bias is learned and amplified when training on biased, machine-generated labels, a critical finding for automated annotation pipelines. This work introduces a systematic framework for diagnosing algorithmic bias in medical segmentation and demonstrates that achieving fairness requires addressing qualitative distributional differences rather than merely balancing case counts.