Confidence-based Out-of-Distribution Detection: A Comparative Study and Analysis

TL;DR

This paper compares various confidence-based out-of-distribution detection methods across general computer vision and medical imaging tasks, revealing that high performance in one domain does not guarantee success in another.

Contribution

It provides a comprehensive comparative analysis of state-of-the-art OOD detection methods and insights into their performance differences between general and medical imaging tasks.

Findings

High performance in computer vision does not ensure accuracy in medical imaging.

Factors influencing OOD detection performance vary between tasks.

The study offers insights for developing improved OOD detection methods.

Abstract

Image classification models deployed in the real world may receive inputs outside the intended data distribution. For critical applications such as clinical decision making, it is important that a model can detect such out-of-distribution (OOD) inputs and express its uncertainty. In this work, we assess the capability of various state-of-the-art approaches for confidence-based OOD detection through a comparative study and in-depth analysis. First, we leverage a computer vision benchmark to reproduce and compare multiple OOD detection methods. We then evaluate their capabilities on the challenging task of disease classification using chest X-rays. Our study shows that high performance in a computer vision task does not directly translate to accuracy in a medical imaging task. We analyse factors that affect performance of the methods between the two tasks. Our results provide useful insights for developing the next generation of OOD detection methods.