Hesitation is defeat? Connecting Linguistic and Predictive Uncertainty

TL;DR

This paper explores the relationship between machine-derived predictive uncertainty and human linguistic uncertainty in chest radiograph interpretation, highlighting the challenges and potential of Bayesian Deep Learning in clinical decision support.

Contribution

It evaluates different Bayesian methods for uncertainty estimation in medical imaging and analyzes their correlation with human uncertainty expressed in radiology reports.

Findings

Good model performance achieved

Modest correlation between predictive and linguistic uncertainty found

Bayesian methods provide valuable but incomplete uncertainty estimates

Abstract

Automating chest radiograph interpretation using Deep Learning (DL) models has the potential to significantly improve clinical workflows, decision-making, and large-scale health screening. However, in medical settings, merely optimising predictive performance is insufficient, as the quantification of uncertainty is equally crucial. This paper investigates the relationship between predictive uncertainty, derived from Bayesian Deep Learning approximations, and human/linguistic uncertainty, as estimated from free-text radiology reports labelled by rule-based labellers. Utilising BERT as the model of choice, this study evaluates different binarisation methods for uncertainty labels and explores the efficacy of Monte Carlo Dropout and Deep Ensembles in estimating predictive uncertainty. The results demonstrate good model performance, but also a modest correlation between predictive and linguistic uncertainty, highlighting the challenges in aligning machine uncertainty with human interpretation nuances. Our findings suggest that while Bayesian approximations provide valuable uncertainty estimates, further refinement is necessary to fully capture and utilise the subtleties of human uncertainty in clinical applications.