Towards unraveling calibration biases in medical image analysis

TL;DR

This paper investigates calibration biases in AI models for medical image analysis, highlighting how sample size disparities can bias fairness assessments and emphasizing the importance of proper calibration evaluation in clinical decision support systems.

Contribution

It reveals systematic biases in calibration metrics related to sample sizes and discusses their impact on fairness analysis in medical AI models.

Findings

Calibration metrics are biased by sample size differences.

Sample imbalance can confound fairness assessments.

Calibration biases affect clinical decision support systems.

Abstract

In recent years the development of artificial intelligence (AI) systems for automated medical image analysis has gained enormous momentum. At the same time, a large body of work has shown that AI systems can systematically and unfairly discriminate against certain populations in various application scenarios. These two facts have motivated the emergence of algorithmic fairness studies in this field. Most research on healthcare algorithmic fairness to date has focused on the assessment of biases in terms of classical discrimination metrics such as AUC and accuracy. Potential biases in terms of model calibration, however, have only recently begun to be evaluated. This is especially important when working with clinical decision support systems, as predictive uncertainty is key for health professionals to optimally evaluate and combine multiple sources of information. In this work we study discrimination and calibration biases in models trained for automatic detection of malignant dermatological conditions from skin lesions images. Importantly, we show how several typically employed calibration metrics are systematically biased with respect to sample sizes, and how this can lead to erroneous fairness analysis if not taken into consideration. This is of particular relevance to fairness studies, where data imbalance results in drastic sample size differences between demographic sub-groups, which, if not taken into account, can act as confounders.