Conserve-Update-Revise to Cure Generalization and Robustness Trade-off in Adversarial Training

TL;DR

This paper introduces CURE, a novel training framework that selectively updates neural network layers to improve robustness against adversarial attacks while maintaining generalization, addressing the robustness-generalization trade-off.

Contribution

The paper proposes CURE, a dataset- and architecture-agnostic method using gradient prominence for selective weight updates to enhance robustness and generalization in adversarial training.

Findings

CURE improves robustness and reduces overfitting.

Selective layer updating enhances learning capacity.

Mitigates robust overfitting across datasets and architectures.

Abstract

Adversarial training improves the robustness of neural networks against adversarial attacks, albeit at the expense of the trade-off between standard and robust generalization. To unveil the underlying factors driving this phenomenon, we examine the layer-wise learning capabilities of neural networks during the transition from a standard to an adversarial setting. Our empirical findings demonstrate that selectively updating specific layers while preserving others can substantially enhance the network's learning capacity. We therefore propose CURE, a novel training framework that leverages a gradient prominence criterion to perform selective conservation, updating, and revision of weights. Importantly, CURE is designed to be dataset- and architecture-agnostic, ensuring its applicability across various scenarios. It effectively tackles both memorization and overfitting issues, thus enhancing the trade-off between robustness and generalization and additionally, this training approach also aids in mitigating "robust overfitting". Furthermore, our study provides valuable insights into the mechanisms of selective adversarial training and offers a promising avenue for future research.