Confidence Intervals for Performance Estimates in Brain MRI Segmentation

TL;DR

This paper investigates how many test images are needed to reliably estimate performance in 3D brain MRI segmentation, showing that fewer samples are needed than in classification tasks, with implications for reporting confidence in medical imaging.

Contribution

It provides an analysis of confidence interval widths in brain MRI segmentation, demonstrating that fewer test samples are sufficient for reliable estimates compared to classification.

Findings

Confidence intervals can be approximated by parametric methods for segmentation.

Fewer test images are needed for a given precision in segmentation than in classification.

Typically, 100-200 samples suffice for a 1% confidence interval width in low-spread scenarios.

Abstract

Medical segmentation models are evaluated empirically. As such an evaluation is based on a limited set of example images, it is unavoidably noisy. Beyond a mean performance measure, reporting confidence intervals is thus crucial. However, this is rarely done in medical image segmentation. The width of the confidence interval depends on the test set size and on the spread of the performance measure (its standard-deviation across the test set). For classification, many test images are needed to avoid wide confidence intervals. Segmentation, however, has not been studied, and it differs by the amount of information brought by a given test image. In this paper, we study the typical confidence intervals in the context of segmentation in 3D brain magnetic resonance imaging (MRI). We carry experiments on using the standard nnU-net framework, two datasets from the Medical Decathlon challenge that concern brain MRI (hippocampus and brain tumor segmentation) and two performance measures: the Dice Similarity Coefficient and the Hausdorff distance. We show that the parametric confidence intervals are reasonable approximations of the bootstrap estimates for varying test set sizes and spread of the performance metric. Importantly, we show that the test size needed to achieve a given precision is often much lower than for classification tasks. Typically, a 1\% wide confidence interval requires about 100-200 test samples when the spread is low (standard-deviation around 3\%). More difficult segmentation tasks may lead to higher spreads and require over 1000 samples.