Efficient Adversarial Training with Robust Early-Bird Tickets

TL;DR

This paper introduces a novel efficient adversarial training method for pre-trained language models that leverages early emergence of robust subnetworks, significantly reducing training time while maintaining robustness.

Contribution

The authors discover robust early-bird tickets in early training phases and develop a structured sparsity search method to accelerate adversarial training.

Findings

Achieves 7 to 13 times faster training speed.

Maintains or improves robustness compared to state-of-the-art methods.

Automatically terminates ticket search with a convergence metric.

Abstract

Adversarial training is one of the most powerful methods to improve the robustness of pre-trained language models (PLMs). However, this approach is typically more expensive than traditional fine-tuning because of the necessity to generate adversarial examples via gradient descent. Delving into the optimization process of adversarial training, we find that robust connectivity patterns emerge in the early training phase (typically $0.15\sim0.3$ epochs), far before parameters converge. Inspired by this finding, we dig out robust early-bird tickets (i.e., subnetworks) to develop an efficient adversarial training method: (1) searching for robust tickets with structured sparsity in the early stage; (2) fine-tuning robust tickets in the remaining time. To extract the robust tickets as early as possible, we design a ticket convergence metric to automatically terminate the searching process. Experiments show that the proposed efficient adversarial training method can achieve up to $7\times \sim 13 \times$ training speedups while maintaining comparable or even better robustness compared to the most competitive state-of-the-art adversarial training methods.