Uncertainty estimation for classification and risk prediction on medical tabular data

TL;DR

This paper enhances uncertainty estimation methods for medical tabular data, focusing on clinical relevance, out-of-domain detection, and the impact of data characteristics, supported by experiments on real and synthetic datasets.

Contribution

It introduces refined heuristics for selecting uncertainty techniques tailored to clinical scenarios and compares ensemble methods with auto-encoders for out-of-domain detection.

Findings

Ensemble methods perform poorly in out-of-domain detection compared to auto-encoders.

Heuristics for uncertainty estimation vary depending on clinical use-case.

Auto-encoders are more effective for detecting out-of-domain examples in medical data.

Abstract

In a data-scarce field such as healthcare, where models often deliver predictions on patients with rare conditions, the ability to measure the uncertainty of a model's prediction could potentially lead to improved effectiveness of decision support tools and increased user trust. This work advances the understanding of uncertainty estimation for classification and risk prediction on medical tabular data, in a two-fold way. First, we expand and refine the set of heuristics to select an uncertainty estimation technique, introducing tests for clinically-relevant scenarios such as generalization to uncommon pathologies, changes in clinical protocol and simulations of corrupted data. We furthermore differentiate these heuristics depending on the clinical use-case. Second, we observe that ensembles and related techniques perform poorly when it comes to detecting out-of-domain examples, a critical task which is carried out more successfully by auto-encoders. These remarks are enriched by considerations of the interplay of uncertainty estimation with class imbalance, post-modeling calibration and other modeling procedures. Our findings are supported by an array of experiments on toy and real-world data.