Masking Adversarial Damage: Finding Adversarial Saliency for Robust and Sparse Network

TL;DR

This paper introduces MAD, a novel adversarial pruning method that uses second-order information to identify and remove non-essential parameters, maintaining robustness while reducing model size.

Contribution

MAD is the first adversarial pruning technique leveraging second-order loss information to preserve robustness and sparsity simultaneously.

Findings

MAD effectively prunes adversarially trained networks without losing robustness.

Model parameters in the initial layer are highly sensitive to adversarial examples.

Compressed feature representations retain semantic information for target objects.

Abstract

Adversarial examples provoke weak reliability and potential security issues in deep neural networks. Although adversarial training has been widely studied to improve adversarial robustness, it works in an over-parameterized regime and requires high computations and large memory budgets. To bridge adversarial robustness and model compression, we propose a novel adversarial pruning method, Masking Adversarial Damage (MAD) that employs second-order information of adversarial loss. By using it, we can accurately estimate adversarial saliency for model parameters and determine which parameters can be pruned without weakening adversarial robustness. Furthermore, we reveal that model parameters of initial layer are highly sensitive to the adversarial examples and show that compressed feature representation retains semantic information for the target objects. Through extensive experiments on three public datasets, we demonstrate that MAD effectively prunes adversarially trained networks without loosing adversarial robustness and shows better performance than previous adversarial pruning methods.