Adversarial Defense in Vision-Language Models: An Overview

TL;DR

This paper reviews recent strategies to defend vision-language models like CLIP against adversarial attacks, categorizing defenses into training-time, test-time adaptation, and training-free methods, and discusses their strengths and limitations.

Contribution

It provides a comprehensive overview of current adversarial defense techniques for VLMs, highlighting recent advancements and ongoing challenges in robustness enhancement.

Findings

Training-time defenses improve robustness but are resource-intensive.

Test-time adaptation offers flexibility but increases complexity.

Training-free methods mitigate attacks without additional training.

Abstract

The widespread use of Vision Language Models (VLMs, e.g. CLIP) has raised concerns about their vulnerability to sophisticated and imperceptible adversarial attacks. These attacks could compromise model performance and system security in cross-modal tasks. To address this challenge, three main defense paradigms have been proposed: Training-time Defense, Test-time Adaptation Defense, and Training-free Defense. Training-time Defense involves modifying the training process, typically through adversarial fine-tuning to improve the robustness to adversarial examples. While effective, this approach requires substantial computational resources and may not generalize across all adversarial attacks. Test-time Adaptation Defense focuses on adapting the model at inference time by updating its parameters to handle unlabeled adversarial examples, offering flexibility but often at the cost of increased complexity and computational overhead. Training-free Defense avoids modifying the model itself, instead focusing on altering the adversarial inputs or their feature embeddings, which enforces input perturbations to mitigate the impact of attacks without additional training. This survey reviews the latest advancements in adversarial defense strategies for VLMs, highlighting the strengths and limitations of such approaches and discussing ongoing challenges in enhancing the robustness of VLMs.