Are Fast Methods Stable in Adversarially Robust Transfer Learning?

TL;DR

This paper investigates the stability and efficiency of using the fast gradient sign method (FGSM) for adversarially robust transfer learning, finding it more stable and faster than PGD with minimal robustness loss.

Contribution

The study reveals that FGSM is more stable and computationally efficient than PGD in adversarial transfer learning, especially with parameter-efficient fine-tuning methods.

Findings

FGSM does not suffer from catastrophic overfitting at standard perturbation budgets.

FGSM remains stable up to higher perturbation levels with parameter-efficient fine-tuning.

FGSM achieves comparable robustness to PGD while requiring four times less training time.

Abstract

Transfer learning is often used to decrease the computational cost of model training, as fine-tuning a model allows a downstream task to leverage the features learned from the pre-training dataset and quickly adapt them to a new task. This is particularly useful for achieving adversarial robustness, as adversarially training models from scratch is very computationally expensive. However, high robustness in transfer learning still requires adversarial training during the fine-tuning phase, which requires up to an order of magnitude more time than standard fine-tuning. In this work, we revisit the use of the fast gradient sign method (FGSM) in robust transfer learning to improve the computational cost of adversarial fine-tuning. We surprisingly find that FGSM is much more stable in adversarial fine-tuning than when training from scratch. In particular, FGSM fine-tuning does not suffer from any issues with catastrophic overfitting at standard perturbation budgets of $\varepsilon=4$ or $\varepsilon=8$. This stability is further enhanced with parameter-efficient fine-tuning methods, where FGSM remains stable even up to $\varepsilon=32$ for linear probing. We demonstrate how this stability translates into performance across multiple datasets. Compared to fine-tuning with the more commonly used method of projected gradient descent (PGD), on average, FGSM only loses 0.39% and 1.39% test robustness for $\varepsilon=4$ and $\varepsilon=8$ while using $4\times$ less training time. Surprisingly, FGSM may not only be a significantly more efficient alternative to PGD in adversarially robust transfer learning but also a well-performing one.