Confidence Matters: Uncertainty Quantification and Precision Assessment of Deep Learning-based CMR Biomarker Estimates Using Scan-rescan Data

TL;DR

This paper evaluates uncertainty estimation techniques in deep learning for cardiac MRI biomarker analysis, emphasizing the importance of distribution-based metrics over traditional accuracy measures for assessing scan-rescan precision.

Contribution

It introduces new distribution-based metrics for biomarker precision assessment and demonstrates their importance in evaluating deep learning model reliability.

Findings

High accuracy (Dice 87%) achieved on external datasets.

Distribution metrics showed less than 45% overlap in confidence intervals.

Significant differences in biomarkers between scan and rescan in over 65% of cases.

Abstract

The performance of deep learning (DL) methods for the analysis of cine cardiovascular magnetic resonance (CMR) is typically assessed in terms of accuracy, overlooking precision. In this work, uncertainty estimation techniques, namely deep ensemble, test-time augmentation, and Monte Carlo dropout, are applied to a state-of-the-art DL pipeline for cardiac functional biomarker estimation, and new distribution-based metrics are proposed for the assessment of biomarker precision. The model achieved high accuracy (average Dice 87%) and point estimate precision on two external validation scan-rescan CMR datasets. However, distribution-based metrics showed that the overlap between scan/rescan confidence intervals was >50% in less than 45% of the cases. Statistical similarity tests between scan and rescan biomarkers also resulted in significant differences for over 65% of the cases. We conclude that, while point estimate metrics might suggest good performance, distributional analyses reveal lower precision, highlighting the need to use more representative metrics to assess scan-rescan agreement.