Deployment of Image Analysis Algorithms under Prevalence Shifts

TL;DR

This paper investigates how prevalence shifts affect medical image classifiers, demonstrating their impact on calibration and decision thresholds, and proposes a prevalence-aware workflow to improve deployment performance without extra annotations.

Contribution

It empirically analyzes the effects of prevalence shifts on classifier calibration and performance metrics, and introduces a workflow to adapt classifiers to new prevalence environments.

Findings

Prevalence shifts cause significant miscalibration and decision threshold deviations.

Validation metrics often fail to reflect true deployment performance under prevalence changes.

The proposed workflow improves classifier decision-making and performance estimation in new environments.

Abstract

Domain gaps are among the most relevant roadblocks in the clinical translation of machine learning (ML)-based solutions for medical image analysis. While current research focuses on new training paradigms and network architectures, little attention is given to the specific effect of prevalence shifts on an algorithm deployed in practice. Such discrepancies between class frequencies in the data used for a method's development/validation and that in its deployment environment(s) are of great importance, for example in the context of artificial intelligence (AI) democratization, as disease prevalences may vary widely across time and location. Our contribution is twofold. First, we empirically demonstrate the potentially severe consequences of missing prevalence handling by analyzing (i) the extent of miscalibration, (ii) the deviation of the decision threshold from the optimum, and (iii) the ability of validation metrics to reflect neural network performance on the deployment population as a function of the discrepancy between development and deployment prevalence. Second, we propose a workflow for prevalence-aware image classification that uses estimated deployment prevalences to adjust a trained classifier to a new environment, without requiring additional annotated deployment data. Comprehensive experiments based on a diverse set of 30 medical classification tasks showcase the benefit of the proposed workflow in generating better classifier decisions and more reliable performance estimates compared to current practice.