Concerning Uncertainty -- A Systematic Survey of Uncertainty-Aware XAI

TL;DR

This survey reviews how uncertainty is integrated into explainable AI, highlighting methods, evaluation practices, and proposing unified principles to improve reliability and user trust.

Contribution

It systematically categorizes uncertainty quantification approaches in XAI and emphasizes the need for standardized evaluation linking uncertainty, robustness, and human decision-making.

Findings

Three main approaches: Bayesian, Monte Carlo, Conformal methods.

Evaluation practices are fragmented and focus mainly on models.

Recent trends favor calibration and distribution-free techniques.

Abstract

This paper surveys uncertainty-aware explainable artificial intelligence (UAXAI), examining how uncertainty is incorporated into explanatory pipelines and how such methods are evaluated. Across the literature, three recurring approaches to uncertainty quantification emerge (Bayesian, Monte Carlo, and Conformal methods), alongside distinct strategies for integrating uncertainty into explanations: assessing trustworthiness, constraining models or explanations, and explicitly communicating uncertainty. Evaluation practices remain fragmented and largely model centered, with limited attention to users and inconsistent reporting of reliability properties (e.g., calibration, coverage, explanation stability). Recent work leans towards calibration, distribution free techniques and recognizes explainer variability as a central concern. We argue that progress in UAXAI requires unified evaluation principles that link uncertainty propagation, robustness, and human decision-making, and highlight counterfactual and calibration approaches as promising avenues for aligning interpretability with reliability.