Detecting when pre-trained nnU-Net models fail silently for Covid-19 lung lesion segmentation

TL;DR

This paper introduces a lightweight Mahalanobis distance-based method for detecting silent failures of pre-trained nnU-Net models in Covid-19 lung lesion segmentation, enhancing trustworthiness in clinical applications.

Contribution

It presents a novel, easily integrable OOD detection technique that identifies when pre-trained models may fail silently on new, out-of-distribution data.

Findings

Effectively detects incorrect segmentations in Covid-19 lung CT scans.

Seamlessly integrates into existing segmentation pipelines without retraining.

Improves reliability of deep learning models in clinical settings.

Abstract

Automatic segmentation of lung lesions in computer tomography has the potential to ease the burden of clinicians during the Covid-19 pandemic. Yet predictive deep learning models are not trusted in the clinical routine due to failing silently in out-of-distribution (OOD) data. We propose a lightweight OOD detection method that exploits the Mahalanobis distance in the feature space. The proposed approach can be seamlessly integrated into state-of-the-art segmentation pipelines without requiring changes in model architecture or training procedure, and can therefore be used to assess the suitability of pre-trained models to new data. We validate our method with a patch-based nnU-Net architecture trained with a multi-institutional dataset and find that it effectively detects samples that the model segments incorrectly.