Improving Fast Adversarial Training Paradigm: An Example Taxonomy Perspective

TL;DR

This paper analyzes the causes of catastrophic overfitting in fast adversarial training (FAT) and proposes a new paradigm, ETA, with loss adaptation and dynamic strategies to improve training robustness and performance.

Contribution

It introduces an example taxonomy for FAT, identifies causes of overfitting, and proposes ETA with loss adaptation and batch momentum to enhance training effectiveness.

Findings

ETA achieves state-of-the-art performance.

Proposed methods effectively prevent catastrophic overfitting.

Experiments validate the competitiveness across datasets.

Abstract

While adversarial training is an effective defense method against adversarial attacks, it notably increases the training cost. To this end, fast adversarial training (FAT) is presented for efficient training and has become a hot research topic. However, FAT suffers from catastrophic overfitting, which leads to a performance drop compared with multi-step adversarial training. However, the cause of catastrophic overfitting remains unclear and lacks exploration. In this paper, we present an example taxonomy in FAT, which identifies that catastrophic overfitting is caused by the imbalance between the inner and outer optimization in FAT. Furthermore, we investigated the impact of varying degrees of training loss, revealing a correlation between training loss and catastrophic overfitting. Based on these observations, we redesign the loss function in FAT with the proposed dynamic label relaxation to concentrate the loss range and reduce the impact of misclassified examples. Meanwhile, we introduce batch momentum initialization to enhance the diversity to prevent catastrophic overfitting in an efficient manner. Furthermore, we also propose Catastrophic Overfitting aware Loss Adaptation (COLA), which employs a separate training strategy for examples based on their loss degree. Our proposed method, named example taxonomy aware FAT (ETA), establishes an improved paradigm for FAT. Experiment results demonstrate our ETA achieves state-of-the-art performance. Comprehensive experiments on four standard datasets demonstrate the competitiveness of our proposed method.