Improving Perturbation-based Explanations by Understanding the Role of Uncertainty Calibration

TL;DR

This paper explores how uncertainty calibration affects the reliability of perturbation-based explanations in machine learning, introducing ReCalX to improve explanation robustness by recalibrating model confidence.

Contribution

It reveals the impact of miscalibration on explanation quality and proposes ReCalX, a novel method for recalibrating models to enhance explanation reliability without altering predictions.

Findings

ReCalX reduces perturbation-specific miscalibration effectively.

ReCalX improves explanation robustness across diverse models.

ReCalX enhances identification of globally important features.

Abstract

Perturbation-based explanations are widely utilized to enhance the transparency of machine-learning models in practice. However, their reliability is often compromised by the unknown model behavior under the specific perturbations used. This paper investigates the relationship between uncertainty calibration - the alignment of model confidence with actual accuracy - and perturbation-based explanations. We show that models systematically produce unreliable probability estimates when subjected to explainability-specific perturbations and theoretically prove that this directly undermines global and local explanation quality. To address this, we introduce ReCalX, a novel approach to recalibrate models for improved explanations while preserving their original predictions. Empirical evaluations across diverse models and datasets demonstrate that ReCalX consistently reduces perturbation-specific miscalibration most effectively while enhancing explanation robustness and the identification of globally important input features.