Explaining Classifiers using Adversarial Perturbations on the Perceptual Ball

TL;DR

This paper introduces a perceptually regularized adversarial perturbation method that produces semi-sparse, semantically meaningful explanations highlighting objects in images, bridging counterfactual explanations and adversarial attacks.

Contribution

It proposes a novel perceptual loss-based regularization for adversarial perturbations that enhances interpretability by focusing on relevant image regions.

Findings

Effective in weak localization benchmarks

Improves insertion and deletion metrics

Enhances pointing game performance

Abstract

We present a simple regularization of adversarial perturbations based upon the perceptual loss. While the resulting perturbations remain imperceptible to the human eye, they differ from existing adversarial perturbations in that they are semi-sparse alterations that highlight objects and regions of interest while leaving the background unaltered. As a semantically meaningful adverse perturbations, it forms a bridge between counterfactual explanations and adversarial perturbations in the space of images. We evaluate our approach on several standard explainability benchmarks, namely, weak localization, insertion deletion, and the pointing game demonstrating that perceptually regularized counterfactuals are an effective explanation for image-based classifiers.