Deep neural network loses attention to adversarial images

TL;DR

This paper investigates how different adversarial attacks affect neural network attention by analyzing saliency and activation maps, revealing diverse mechanisms of attention disruption and implications for robustness.

Contribution

It provides a detailed analysis of how various adversarial attacks distort neural network attention, enhancing understanding of attack mechanisms and defense vulnerabilities.

Findings

Pixel Attack redirects attention to perturbed pixels or away from them

Projected Gradient Descent attack reduces attention in intermediate layers

Different attacks impact saliency and activation maps in distinct ways

Abstract

Adversarial algorithms have shown to be effective against neural networks for a variety of tasks. Some adversarial algorithms perturb all the pixels in the image minimally for the image classification task in image classification. In contrast, some algorithms perturb few pixels strongly. However, very little information is available regarding why these adversarial samples so diverse from each other exist. Recently, Vargas et al. showed that the existence of these adversarial samples might be due to conflicting saliency within the neural network. We test this hypothesis of conflicting saliency by analysing the Saliency Maps (SM) and Gradient-weighted Class Activation Maps (Grad-CAM) of original and few different types of adversarial samples. We also analyse how different adversarial samples distort the attention of the neural network compared to original samples. We show that in the case of Pixel Attack, perturbed pixels either calls the network attention to themselves or divert the attention from them. Simultaneously, the Projected Gradient Descent Attack perturbs pixels so that intermediate layers inside the neural network lose attention for the correct class. We also show that both attacks affect the saliency map and activation maps differently. Thus, shedding light on why some defences successful against some attacks remain vulnerable against other attacks. We hope that this analysis will improve understanding of the existence and the effect of adversarial samples and enable the community to develop more robust neural networks.