Understanding Robustness of Parameter-Efficient Tuning for Image Classification

TL;DR

This paper systematically investigates the robustness of parameter-efficient tuning methods for image classification against various adversarial and information perturbation attacks, providing insights for improving their reliability.

Contribution

It offers a comprehensive analysis of four classical PET techniques under diverse attack scenarios, which was lacking in prior research.

Findings

PET methods show varying robustness to white-box attacks.

Transferability of adversarial samples affects PET robustness.

Information perturbations significantly impact PET performance.

Abstract

Parameter-efficient tuning (PET) techniques calibrate the model's predictions on downstream tasks by freezing the pre-trained models and introducing a small number of learnable parameters. However, despite the numerous PET methods proposed, their robustness has not been thoroughly investigated. In this paper, we systematically explore the robustness of four classical PET techniques (e.g., VPT, Adapter, AdaptFormer, and LoRA) under both white-box attacks and information perturbations. For white-box attack scenarios, we first analyze the performance of PET techniques using FGSM and PGD attacks. Subsequently, we further explore the transferability of adversarial samples and the impact of learnable parameter quantities on the robustness of PET methods. Under information perturbation attacks, we introduce four distinct perturbation strategies, including Patch-wise Drop, Pixel-wise Drop, Patch Shuffle, and Gaussian Noise, to comprehensively assess the robustness of these PET techniques in the presence of information loss. Via these extensive studies, we enhance the understanding of the robustness of PET methods, providing valuable insights for improving their performance in computer vision applications. The code is available at https://github.com/JCruan519/PETRobustness.