Does deep learning model calibration improve performance in class-imbalanced medical image classification?

TL;DR

This study systematically analyzes how model calibration affects deep learning performance in class-imbalanced medical image classification, showing calibration improves results at default thresholds but not at optimal thresholds.

Contribution

It provides a comprehensive analysis of calibration effects across different medical imaging modalities, imbalance levels, and thresholds, clarifying when calibration is beneficial.

Findings

Calibration improves performance at default threshold of 0.5.

No significant benefit of calibration at PR-guided threshold.

Effects consistent across chest X-ray and fundus image datasets.

Abstract

In medical image classification tasks, it is common to find that the number of normal samples far exceeds the number of abnormal samples. In such class-imbalanced situations, reliable training of deep neural networks continues to be a major challenge. Under these circumstances, the predicted class probabilities may be biased toward the majority class. Calibration has been suggested to alleviate some of these effects. However, there is insufficient analysis explaining when and whether calibrating a model would be beneficial in improving performance. In this study, we perform a systematic analysis of the effect of model calibration on its performance on two medical image modalities, namely, chest X-rays and fundus images, using various deep learning classifier backbones. For this, we study the following variations: (i) the degree of imbalances in the dataset used for training; (ii) calibration methods; and (iii) two classification thresholds, namely, default decision threshold of 0.5, and optimal threshold from precision-recall curves. Our results indicate that at the default operating threshold of 0.5, the performance achieved through calibration is significantly superior (p < 0.05) to using uncalibrated probabilities. However, at the PR-guided threshold, these gains are not significantly different (p > 0.05). This finding holds for both image modalities and at varying degrees of imbalance.