DIAGNOSE: Avoiding Out-of-distribution Data using Submodular Information Measures

TL;DR

Diagnose is an active learning framework that effectively distinguishes in-distribution from out-of-distribution data in medical imaging, reducing labeling costs and improving model performance by leveraging submodular information measures.

Contribution

The paper introduces a novel active learning method that jointly models similarity and dissimilarity to better avoid OOD data in medical imaging tasks.

Findings

Diagnose outperforms existing active learning methods in medical imaging scenarios.

The framework effectively reduces OOD data inclusion during training.

Experiments demonstrate improved model accuracy and reduced labeling costs.

Abstract

Avoiding out-of-distribution (OOD) data is critical for training supervised machine learning models in the medical imaging domain. Furthermore, obtaining labeled medical data is difficult and expensive since it requires expert annotators like doctors, radiologists, etc. Active learning (AL) is a well-known method to mitigate labeling costs by selecting the most diverse or uncertain samples. However, current AL methods do not work well in the medical imaging domain with OOD data. We propose Diagnose (avoiDing out-of-dIstribution dAta usinG submodular iNfOrmation meaSurEs), a novel active learning framework that can jointly model similarity and dissimilarity, which is crucial in mining in-distribution data and avoiding OOD data at the same time. Particularly, we use a small number of data points as exemplars that represent a query set of in-distribution data points and a private set of OOD data points. We illustrate the generalizability of our framework by evaluating it on a wide variety of real-world OOD scenarios. Our experiments verify the superiority of Diagnose over the state-of-the-art AL methods across multiple domains of medical imaging.