Text-Guided Attention is All You Need for Zero-Shot Robustness in Vision-Language Models

TL;DR

This paper introduces TGA-ZSR, a novel framework that improves the adversarial robustness of vision-language models like CLIP by aligning and constraining text-guided attention, resulting in significant accuracy gains across multiple datasets.

Contribution

The paper proposes a new attention-based method, TGA-ZSR, that enhances zero-shot robustness of CLIP models by aligning and constraining text-guided attention with original clean examples.

Findings

Achieves 9.58% improvement in zero-shot robust accuracy.

Validates effectiveness across 16 datasets.

Introduces attention refinement and model constraint modules.

Abstract

Due to the impressive zero-shot capabilities, pre-trained vision-language models (e.g. CLIP), have attracted widespread attention and adoption across various domains. Nonetheless, CLIP has been observed to be susceptible to adversarial examples. Through experimental analysis, we have observed a phenomenon wherein adversarial perturbations induce shifts in text-guided attention. Building upon this observation, we propose a simple yet effective strategy: Text-Guided Attention for Zero-Shot Robustness (TGA-ZSR). This framework incorporates two components: the Attention Refinement module and the Attention-based Model Constraint module. Our goal is to maintain the generalization of the CLIP model and enhance its adversarial robustness: The Attention Refinement module aligns the text-guided attention obtained from the target model via adversarial examples with the text-guided attention acquired from the original model via clean examples. This alignment enhances the model's robustness. Additionally, the Attention-based Model Constraint module acquires text-guided attention from both the target and original models using clean examples. Its objective is to maintain model performance on clean samples while enhancing overall robustness. The experiments validate that our method yields a 9.58% enhancement in zero-shot robust accuracy over the current state-of-the-art techniques across 16 datasets. Our code is available at https://github.com/zhyblue424/TGA-ZSR.