SAFER: Sharpness Aware layer-selective Finetuning for Enhanced Robustness in vision transformers

TL;DR

SAFER is a novel layer-selective fine-tuning method for vision transformers that improves robustness against adversarial attacks by focusing on vulnerable layers with sharpness-aware minimization.

Contribution

The paper introduces SAFER, a layer-selective fine-tuning approach that mitigates adversarial overfitting in ViTs by selectively optimizing vulnerable layers.

Findings

Enhances clean and adversarial accuracy by around 5% on average.

Achieves up to 20% improvement in certain ViT architectures.

Effective across various datasets and model architectures.

Abstract

Vision transformers (ViTs) have become essential backbones in advanced computer vision applications and multi-modal foundation models. Despite their strengths, ViTs remain vulnerable to adversarial perturbations, comparable to or even exceeding the vulnerability of convolutional neural networks (CNNs). Furthermore, the large parameter count and complex architecture of ViTs make them particularly prone to adversarial overfitting, often compromising both clean and adversarial accuracy. This paper mitigates adversarial overfitting in ViTs through a novel, layer-selective fine-tuning approach: SAFER. Instead of optimizing the entire model, we identify and selectively fine-tune a small subset of layers most susceptible to overfitting, applying sharpness-aware minimization to these layers while freezing the rest of the model. Our method consistently enhances both clean and adversarial accuracy over baseline approaches. Typical improvements are around 5%, with some cases achieving gains as high as 20% across various ViT architectures and datasets.