Limitations of Out-of-Distribution Detection in 3D Medical Image Segmentation

TL;DR

This paper evaluates the performance of existing out-of-distribution detection methods in 3D medical image segmentation, revealing significant limitations and proposing a simple, effective baseline method called IHF, along with new challenges for benchmarking.

Contribution

It introduces new OOD challenges for 3D medical segmentation, demonstrates the limitations of current methods, and proposes IHF as a competitive baseline, advancing research in this area.

Findings

Existing OOD methods perform poorly on 3D medical segmentation.

Segmentation-specific methods outperform general ones but still have high false positive rates.

IHF achieves better or comparable results, highlighting potential for improvement.

Abstract

Deep Learning models perform unreliably when the data comes from a distribution different from the training one. In critical applications such as medical imaging, out-of-distribution (OOD) detection methods help to identify such data samples, preventing erroneous predictions. In this paper, we further investigate the OOD detection effectiveness when applied to 3D medical image segmentation. We design several OOD challenges representing clinically occurring cases and show that none of these methods achieve acceptable performance. Methods not dedicated to segmentation severely fail to perform in the designed setups; their best mean false positive rate at 95% true positive rate (FPR) is 0.59. Segmentation-dedicated ones still achieve suboptimal performance, with the best mean FPR of 0.31 (lower is better). To indicate this suboptimality, we develop a simple method called Intensity Histogram Features (IHF), which performs comparable or better in the same challenges, with a mean FPR of 0.25. Our findings highlight the limitations of the existing OOD detection methods on 3D medical images and present a promising avenue for improving them. To facilitate research in this area, we release the designed challenges as a publicly available benchmark and formulate practical criteria to test the OOD detection generalization beyond the suggested benchmark. We also propose IHF as a solid baseline to contest the emerging methods.