Hyper Adversarial Tuning for Boosting Adversarial Robustness of Pretrained Large Vision Models

TL;DR

This paper introduces Hyper Adversarial Tuning (HyperAT), a novel method that leverages shared knowledge and hypernetworks to improve the adversarial robustness of large vision models efficiently, surpassing existing approaches.

Contribution

HyperAT is a new framework that combines multiple defense strategies via hypernetworks and knowledge transfer to boost robustness of large vision models.

Findings

HyperAT achieves state-of-the-art robustness on various datasets.

It significantly reduces computational costs compared to traditional adversarial training.

HyperAT outperforms existing fine-tuning methods in robustness metrics.

Abstract

Large vision models have been found vulnerable to adversarial examples, emphasizing the need for enhancing their adversarial robustness. While adversarial training is an effective defense for deep convolutional models, it often faces scalability issues with large vision models due to high computational costs. Recent approaches propose robust fine-tuning methods, such as adversarial tuning of low-rank adaptation (LoRA) in large vision models, but they still struggle to match the accuracy of full parameter adversarial fine-tuning. The integration of various defense mechanisms offers a promising approach to enhancing the robustness of large vision models, yet this paradigm remains underexplored. To address this, we propose hyper adversarial tuning (HyperAT), which leverages shared defensive knowledge among different methods to improve model robustness efficiently and effectively simultaneously. Specifically, adversarial tuning of each defense method is formulated as a learning task, and a hypernetwork generates LoRA specific to this defense. Then, a random sampling and tuning strategy is proposed to extract and facilitate the defensive knowledge transfer between different defenses. Finally, diverse LoRAs are merged to enhance the adversarial robustness. Experiments on various datasets and model architectures demonstrate that HyperAT significantly enhances the adversarial robustness of pretrained large vision models without excessive computational overhead, establishing a new state-of-the-art benchmark.