What Makes VLMs Robust? Towards Reconciling Robustness and Accuracy in Vision-Language Models

TL;DR

This paper investigates the internal mechanisms of adversarial robustness in Vision-Language Models, revealing that robustness is mainly localized in shallow layers and proposing R-Adapt to enhance robustness with minimal modifications.

Contribution

It uncovers the layer-wise distribution of robustness in VLMs and introduces R-Adapt, a simple framework that improves robustness while maintaining accuracy.

Findings

Robustness is primarily localized in shallow layers of VLMs.

Deep layer updates tend to harm both accuracy and robustness.

R-Adapt achieves state-of-the-art robustness on 18 datasets and generalizes to large models.

Abstract

Achieving adversarial robustness in Vision-Language Models (VLMs) inevitably compromises accuracy on clean data, presenting a long-standing and challenging trade-off. In this work, we revisit this trade-off by investigating a fundamental question: What makes VLMs robust? Through a detailed analysis of adversarially fine-tuned models, we examine how robustness mechanisms function internally and how they interact with clean accuracy. Our analysis reveals that adversarial robustness is not uniformly distributed across network depth. Instead, unexpectedly, it is primarily localized within the shallow layers, driven by a low-frequency spectral bias and input-insensitive attention patterns. Meanwhile, updates to the deep layers tend to undermine both clean accuracy and robust generalization. Motivated by these insights, we propose Adversarial Robustness Adaptation (R-Adapt), a simple yet effective framework that freezes all pre-trained weights and introduces minimal, insight-driven adaptations only in the initial layers. This design achieves an exceptional balance between adversarial robustness and clean accuracy. R-Adapt further supports training-free, model-guided, and data-driven paradigms, offering flexible pathways to seamlessly equip standard models with robustness. Extensive evaluations on 18 datasets and diverse tasks demonstrate our state-of-the-art performance under various attacks. Notably, R-Adapt generalizes efficiently to large vision-language models (e.g., LLaVA and Qwen-VL) to enhance their robustness. Our project page is available at https://summu77.github.io/R-Adapt.