Fast Adversarial Training with Smooth Convergence

TL;DR

This paper introduces ConvergeSmooth, a novel oscillatory constraint that stabilizes fast adversarial training by ensuring smooth loss convergence, effectively preventing catastrophic overfitting and enhancing robustness across datasets.

Contribution

The paper proposes ConvergeSmooth, a new method to stabilize FAT by controlling loss convergence, which is attack-agnostic and improves training stability and robustness.

Findings

ConvergeSmooth effectively prevents catastrophic overfitting.

The method improves robustness on multiple datasets.

It outperforms previous FAT techniques in experiments.

Abstract

Fast adversarial training (FAT) is beneficial for improving the adversarial robustness of neural networks. However, previous FAT work has encountered a significant issue known as catastrophic overfitting when dealing with large perturbation budgets, \ie the adversarial robustness of models declines to near zero during training. To address this, we analyze the training process of prior FAT work and observe that catastrophic overfitting is accompanied by the appearance of loss convergence outliers. Therefore, we argue a moderately smooth loss convergence process will be a stable FAT process that solves catastrophic overfitting. To obtain a smooth loss convergence process, we propose a novel oscillatory constraint (dubbed ConvergeSmooth) to limit the loss difference between adjacent epochs. The convergence stride of ConvergeSmooth is introduced to balance convergence and smoothing. Likewise, we design weight centralization without introducing additional hyperparameters other than the loss balance coefficient. Our proposed methods are attack-agnostic and thus can improve the training stability of various FAT techniques. Extensive experiments on popular datasets show that the proposed methods efficiently avoid catastrophic overfitting and outperform all previous FAT methods. Code is available at \url{https://github.com/FAT-CS/ConvergeSmooth}.