A systematic evaluation of uncertainty quantification techniques in deep learning: a case study in photoplethysmography signal analysis

TL;DR

This paper systematically compares eight uncertainty quantification techniques in deep learning models trained on photoplethysmography data for clinical prediction tasks, revealing that the optimal method depends on specific evaluation criteria and practical use cases.

Contribution

It introduces a comprehensive evaluation framework for uncertainty quantification methods in medical time-series analysis, highlighting the importance of local calibration and task-specific assessment.

Findings

Uncertainty technique effectiveness varies with evaluation metrics and reliability scales.

Local calibration and adaptivity provide valuable insights into model behavior.

Evaluation criteria should align with practical clinical use cases.

Abstract

In principle, deep learning models trained on medical time-series, including wearable photoplethysmography (PPG) sensor data, can provide a means to continuously monitor physiological parameters outside of clinical settings. However, there is considerable risk of poor performance when deployed in practical measurement scenarios leading to negative patient outcomes. Reliable uncertainties accompanying predictions can provide guidance to clinicians in their interpretation of the trustworthiness of model outputs. It is therefore of interest to compare the effectiveness of different approaches. Here we implement an unprecedented set of eight uncertainty quantification (UQ) techniques to models trained on two clinically relevant prediction tasks: Atrial Fibrillation (AF) detection (classification), and two variants of blood pressure regression. We formulate a comprehensive evaluation procedure to enable a rigorous comparison of these approaches. We observe a complex picture of uncertainty reliability across the different techniques, where the most optimal for a given task depends on the chosen expression of uncertainty, evaluation metric, and scale of reliability assessed. We find that assessing local calibration and adaptivity provides practically relevant insights about model behaviour that otherwise cannot be acquired using more commonly implemented global reliability metrics. We emphasise that criteria for evaluating UQ techniques should cater to the model's practical use case, where the use of a small number of measurements per patient places a premium on achieving small-scale reliability for the chosen expression of uncertainty, while preserving as much predictive performance as possible.