Robust Mitigation of Age-Dependent Confounding Effects via Sample-Difficulty Decorrelation

TL;DR

This paper introduces a novel method to mitigate age-related confounding in medical image classification by decorrelating age from sample difficulty, improving fairness without sacrificing diagnostic accuracy.

Contribution

The authors propose a robust, sample-difficulty decorrelation framework that targets spurious age-linked trends, preserving meaningful age information while reducing disparities.

Findings

Reduces age-dependent true and false positive disparities

Maintains high AUC performance

Remains robust under train-test age distribution shifts

Abstract

Age dependent performance disparities in medical image classification often arise because age acts as a confounder, linking imaging morphology with disease prevalence. In practice, disparities can manifest as overdiagnosis at ages where disease prevalence is higher and underdiagnosis at ages where prevalence is lower, and can worsen under train test shifts in the age distribution. Conventional mitigation approaches that enforce strict age invariance may suppress diagnostically meaningful information encoded in age. We therefore propose a robust framework that mitigates the effects of age-dependent confounding by targeting spurious age linked trends rather than enforcing invariance. Following a warm-up phase, we characterize sample difficulty and model its age-dependent trends in a label-conditioned manner. We decorrelate age from dominant age difficulty trends using robust, Huber weighted affinity weights, attenuating confounding-driven shortcuts while preserving clinically meaningful, nonlinear age information. We further introduce an Age Coverage Score that scales the decorrelation penalty by minibatch age variance to ensure stable optimization under limited age diversity. Across two radiology datasets, our approach reduces age dependent true and false positive disparities with minimal AUC impact and remains robust to increasing train test age distribution shifts.