On the Connection Between Adversarial Robustness and Saliency Map Interpretability

TL;DR

This paper explores the link between adversarial robustness and saliency map interpretability in neural networks, showing that increased robustness correlates with better alignment between inputs and saliency maps, especially in linear models.

Contribution

It provides a theoretical analysis of the connection in linear models and empirical evidence on neural networks trained with Lipschitz regularization, highlighting where non-linearity affects this relationship.

Findings

Alignment increases with distance from decision boundary in linear models

Robust models have more interpretable saliency maps

Non-linearity weakens the alignment in neural networks

Abstract

Recent studies on the adversarial vulnerability of neural networks have shown that models trained to be more robust to adversarial attacks exhibit more interpretable saliency maps than their non-robust counterparts. We aim to quantify this behavior by considering the alignment between input image and saliency map. We hypothesize that as the distance to the decision boundary grows,so does the alignment. This connection is strictly true in the case of linear models. We confirm these theoretical findings with experiments based on models trained with a local Lipschitz regularization and identify where the non-linear nature of neural networks weakens the relation.