Towards the Memorization Effect of Neural Networks in Adversarial Training

TL;DR

This paper investigates the role of memorization in adversarial training of neural networks, revealing that memorizing atypical samples has limited benefits for robustness and can harm performance on typical samples, leading to a new training method.

Contribution

It introduces Benign Adversarial Training (BAT), a novel approach to improve the trade-off between accuracy and robustness by avoiding harmful memorization during adversarial training.

Findings

Memorizing atypical samples improves accuracy on atypical data but not robustness.

Memorizing certain atypical samples can reduce performance on typical samples.

BAT achieves better accuracy-robustness trade-offs on CIFAR100 and Tiny ImageNet.

Abstract

Recent studies suggest that ``memorization'' is one important factor for overparameterized deep neural networks (DNNs) to achieve optimal performance. Specifically, the perfectly fitted DNNs can memorize the labels of many atypical samples, generalize their memorization to correctly classify test atypical samples and enjoy better test performance. While, DNNs which are optimized via adversarial training algorithms can also achieve perfect training performance by memorizing the labels of atypical samples, as well as the adversarially perturbed atypical samples. However, adversarially trained models always suffer from poor generalization, with both relatively low clean accuracy and robustness on the test set. In this work, we study the effect of memorization in adversarial trained DNNs and disclose two important findings: (a) Memorizing atypical samples is only effective to improve DNN's accuracy on clean atypical samples, but hardly improve their adversarial robustness and (b) Memorizing certain atypical samples will even hurt the DNN's performance on typical samples. Based on these two findings, we propose Benign Adversarial Training (BAT) which can facilitate adversarial training to avoid fitting ``harmful'' atypical samples and fit as more ``benign'' atypical samples as possible. In our experiments, we validate the effectiveness of BAT, and show it can achieve better clean accuracy vs. robustness trade-off than baseline methods, in benchmark datasets such as CIFAR100 and Tiny~ImageNet.