CASE: Contrastive Activation for Saliency Estimation

TL;DR

This paper introduces CASE, a contrastive explanation method for saliency estimation that improves class-specific feature attribution, addressing the limitations of existing saliency methods which often produce class-insensitive explanations.

Contribution

The paper presents a diagnostic test for class sensitivity and proposes CASE, a novel contrastive saliency method that yields more faithful, class-specific explanations.

Findings

Many saliency methods are class-insensitive across architectures and datasets.

CASE produces more faithful and class-specific explanations.

Existing methods often fail to distinguish between competing classes.

Abstract

Saliency methods are widely used to visualize which input features are deemed relevant to a model's prediction. However, their visual plausibility can obscure critical limitations. In this work, we propose a diagnostic test for class sensitivity: a method's ability to distinguish between competing class labels on the same input. Through extensive experiments, we show that many widely used saliency methods produce nearly identical explanations regardless of the class label, calling into question their reliability. We find that class-insensitive behavior persists across architectures and datasets, suggesting the failure mode is structural rather than model-specific. Motivated by these findings, we introduce CASE, a contrastive explanation method that isolates features uniquely discriminative for the predicted class. We evaluate CASE using the proposed diagnostic and a perturbation-based fidelity test, and show that it produces faithful and more class-specific explanations than existing methods.