Why Uncertainty Calibration Matters for Reliable Perturbation-based Explanations

TL;DR

This paper highlights the importance of uncertainty calibration for reliable perturbation-based explanations in machine learning, introduces ReCalX to improve calibration, and demonstrates enhanced explanation quality in vision models.

Contribution

It presents ReCalX, a novel calibration method that improves the reliability of explanations without altering model predictions.

Findings

ReCalX improves explanation alignment with human perception.

Calibration enhances the reliability of perturbation-based explanations.

Models with ReCalX produce more accurate explanations of object locations.

Abstract

Perturbation-based explanations are widely utilized to enhance the transparency of modern machine-learning models. However, their reliability is often compromised by the unknown model behavior under the specific perturbations used. This paper investigates the relationship between uncertainty calibration - the alignment of model confidence with actual accuracy - and perturbation-based explanations. We show that models frequently produce unreliable probability estimates when subjected to explainability-specific perturbations and theoretically prove that this directly undermines explanation quality. To address this, we introduce ReCalX, a novel approach to recalibrate models for improved perturbation-based explanations while preserving their original predictions. Experiments on popular computer vision models demonstrate that our calibration strategy produces explanations that are more aligned with human perception and actual object locations.