Stable and Efficient Adversarial Training through Local Linearization

TL;DR

This paper introduces SEAT, a novel single-step adversarial training method that effectively prevents catastrophic overfitting, leading to robust models with high accuracy and significantly reduced computational costs.

Contribution

SEAT is a new adversarial training approach that mitigates overfitting by leveraging local properties, with strong theoretical support and superior empirical performance.

Findings

SEAT achieves 51% robust accuracy on CIFAR-10 with $l_ive$ perturbations.

SEAT matches 10-step iterative training robustness at only 3% of the computational cost.

Theoretical analysis shows SEAT promotes smooth empirical risk for robustness.

Abstract

There has been a recent surge in single-step adversarial training as it shows robustness and efficiency. However, a phenomenon referred to as ``catastrophic overfitting" has been observed, which is prevalent in single-step defenses and may frustrate attempts to use FGSM adversarial training. To address this issue, we propose a novel method, Stable and Efficient Adversarial Training (SEAT), which mitigates catastrophic overfitting by harnessing on local properties that distinguish a robust model from that of a catastrophic overfitted model. The proposed SEAT has strong theoretical justifications, in that minimizing the SEAT loss can be shown to favour smooth empirical risk, thereby leading to robustness. Experimental results demonstrate that the proposed method successfully mitigates catastrophic overfitting, yielding superior performance amongst efficient defenses. Our single-step method can reach 51% robust accuracy for CIFAR-10 with $l_\infty$ perturbations of radius $8/255$ under a strong PGD-50 attack, matching the performance of a 10-step iterative adversarial training at merely 3% computational cost.