Efficient Adversarial Training With Data Pruning

TL;DR

This paper introduces data pruning as an efficient method to improve adversarial training of neural networks, reducing training time and sometimes enhancing robustness against adversarial attacks.

Contribution

The paper proposes data pruning to increase the efficiency of adversarial training, demonstrating empirical benefits in convergence, reliability, and sometimes accuracy and training speed.

Findings

Data pruning reduces training time significantly.

Random sub-sampling causes some accuracy loss.

In certain cases, data pruning improves both accuracy and efficiency.

Abstract

Neural networks are susceptible to adversarial examples-small input perturbations that cause models to fail. Adversarial training is one of the solutions that stops adversarial examples; models are exposed to attacks during training and learn to be resilient to them. Yet, such a procedure is currently expensive-it takes a long time to produce and train models with adversarial samples, and, what is worse, it occasionally fails. In this paper we demonstrate data pruning-a method for increasing adversarial training efficiency through data sub-sampling.We empirically show that data pruning leads to improvements in convergence and reliability of adversarial training, albeit with different levels of utility degradation. For example, we observe that using random sub-sampling of CIFAR10 to drop 40% of data, we lose 8% adversarial accuracy against the strongest attackers, while by using only 20% of data we lose 14% adversarial accuracy and reduce runtime by a factor of 3. Interestingly, we discover that in some settings data pruning brings benefits from both worlds-it both improves adversarial accuracy and training time.