Assessing the Noise Robustness of Class Activation Maps: A Framework for Reliable Model Interpretability

TL;DR

This paper evaluates the robustness of Class Activation Maps (CAMs) against noise, introduces a new metric for measuring their stability, and demonstrates variability in noise sensitivity across models and datasets.

Contribution

It provides the first comprehensive analysis of CAM noise robustness, proposing a novel metric to quantify stability and sensitivity of CAM explanations under noise perturbations.

Findings

CAMs show significant variability in noise sensitivity.

The proposed robustness metric effectively captures CAM stability.

Empirical evaluation across multiple models and datasets validates the metric.

Abstract

Class Activation Maps (CAMs) are one of the important methods for visualizing regions used by deep learning models. Yet their robustness to different noise remains underexplored. In this work, we evaluate and report the resilience of various CAM methods for different noise perturbations across multiple architectures and datasets. By analyzing the influence of different noise types on CAM explanations, we assess the susceptibility to noise and the extent to which dataset characteristics may impact explanation stability. The findings highlight considerable variability in noise sensitivity for various CAMs. We propose a robustness metric for CAMs that captures two key properties: consistency and responsiveness. Consistency reflects the ability of CAMs to remain stable under input perturbations that do not alter the predicted class, while responsiveness measures the sensitivity of CAMs to changes in the prediction caused by such perturbations. The metric is evaluated empirically across models, different perturbations, and datasets along with complementary statistical tests to exemplify the applicability of our proposed approach.