Does Your Dermatology Classifier Know What It Doesn't Know? Detecting the Long-Tail of Unseen Conditions

TL;DR

This paper presents a deep learning system for skin condition classification that can detect rare and unseen conditions by framing it as an out-of-distribution detection problem, improving clinical safety.

Contribution

The paper introduces hierarchical outlier detection (HOD), a novel approach that jointly performs coarse and fine-grained classification to identify unseen skin conditions.

Findings

HOD improves out-of-distribution detection accuracy.

Ensembling strategies further enhance detection performance.

The framework demonstrates clinical relevance through a cost metric comparison.

Abstract

We develop and rigorously evaluate a deep learning based system that can accurately classify skin conditions while detecting rare conditions for which there is not enough data available for training a confident classifier. We frame this task as an out-of-distribution (OOD) detection problem. Our novel approach, hierarchical outlier detection (HOD) assigns multiple abstention classes for each training outlier class and jointly performs a coarse classification of inliers vs. outliers, along with fine-grained classification of the individual classes. We demonstrate the effectiveness of the HOD loss in conjunction with modern representation learning approaches (BiT, SimCLR, MICLe) and explore different ensembling strategies for further improving the results. We perform an extensive subgroup analysis over conditions of varying risk levels and different skin types to investigate how the OOD detection performance changes over each subgroup and demonstrate the gains of our framework in comparison to baselines. Finally, we introduce a cost metric to approximate downstream clinical impact. We use this cost metric to compare the proposed method against a baseline system, thereby making a stronger case for the overall system effectiveness in a real-world deployment scenario.