Benchmarking Uncertainty Quantification on Biosignal Classification Tasks under Dataset Shift

TL;DR

This paper benchmarks the effectiveness of various uncertainty quantification methods in biosignal classification tasks under dataset shift, revealing limitations in current models' reliability and calibration.

Contribution

It introduces a framework for evaluating uncertainty estimates in biosignal classifiers facing dataset shifts, and benchmarks five methods across multiple tasks.

Findings

Ensemble and Bayesian models perform better in uncertainty estimation under dataset shift.

All tested models show deficiencies in trustworthiness and calibration.

The framework enables comprehensive evaluation of biosignal classifiers' uncertainty estimates.

Abstract

A biosignal is a signal that can be continuously measured from human bodies, such as respiratory sounds, heart activity (ECG), brain waves (EEG), etc, based on which, machine learning models have been developed with very promising performance for automatic disease detection and health status monitoring. However, dataset shift, i.e., data distribution of inference varies from the distribution of the training, is not uncommon for real biosignal-based applications. To improve the robustness, probabilistic models with uncertainty quantification are adapted to capture how reliable a prediction is. Yet, assessing the quality of the estimated uncertainty remains a challenge. In this work, we propose a framework to evaluate the capability of the estimated uncertainty in capturing different types of biosignal dataset shifts with various degrees. In particular, we use three classification tasks based on respiratory sounds and electrocardiography signals to benchmark five representative uncertainty quantification methods. Extensive experiments show that, although Ensemble and Bayesian models could provide relatively better uncertainty estimations under dataset shifts, all tested models fail to meet the promise in trustworthy prediction and model calibration. Our work paves the way for a comprehensive evaluation for any newly developed biosignal classifiers.