Monte Carlo dropout increases model repeatability

TL;DR

This study demonstrates that applying Monte Carlo dropout during testing enhances the repeatability of medical image analysis models across various tasks, crucial for clinical reliability.

Contribution

It provides a comprehensive evaluation of how Monte Carlo dropout improves model repeatability in medical imaging tasks, addressing a key robustness aspect often overlooked.

Findings

Monte Carlo dropout significantly increases model repeatability.

Repeatability improvements observed across all tested tasks.

Average reduction of 17% points in limits of agreement.

Abstract

The integration of artificial intelligence into clinical workflows requires reliable and robust models. Among the main features of robustness is repeatability. Much attention is given to classification performance without assessing the model repeatability, leading to the development of models that turn out to be unusable in practice. In this work, we evaluate the repeatability of four model types on images from the same patient that were acquired during the same visit. We study the performance of binary, multi-class, ordinal, and regression models on three medical image analysis tasks: cervical cancer screening, breast density estimation, and retinopathy of prematurity classification. Moreover, we assess the impact of sampling Monte Carlo dropout predictions at test time on classification performance and repeatability. Leveraging Monte Carlo predictions significantly increased repeatability for all tasks on the binary, multi-class, and ordinal models leading to an average reduction of the 95% limits of agreement by 17% points.