Multimodal Adversarial Defense for Vision-Language Models by Leveraging One-To-Many Relationships

TL;DR

This paper introduces a novel multimodal adversarial training method to defend vision-language models against attacks on both image and text modalities, leveraging one-to-many relationships to improve robustness.

Contribution

It pioneers defense strategies against multimodal attacks in vision-language models and explores the use of diverse, well-aligned data augmentation leveraging one-to-many relationships.

Findings

Multimodal adversarial training significantly outperforms unimodal defenses.

Leveraging one-to-many relationships enhances robustness with well-aligned, diverse data.

Proper augmentation avoids distribution shift, improving defense effectiveness.

Abstract

Pre-trained vision-language (VL) models are highly vulnerable to adversarial attacks. However, existing defense methods primarily focus on image classification, overlooking two key aspects of VL tasks: multimodal attacks, where both image and text can be perturbed, and the one-to-many relationship of images and texts, where a single image can correspond to multiple textual descriptions and vice versa (1:N and N:1). This work is the first to explore defense strategies against multimodal attacks in VL tasks, whereas prior VL defense methods focus on vision robustness. We propose multimodal adversarial training (MAT), which incorporates adversarial perturbations in both image and text modalities during training, significantly outperforming existing unimodal defenses. Furthermore, we discover that MAT is limited by deterministic one-to-one (1:1) image-text pairs in VL training data. To address this, we conduct a comprehensive study on leveraging one-to-many relationships to enhance robustness, investigating diverse augmentation techniques. Our analysis shows that, for a more effective defense, augmented image-text pairs should be well-aligned, diverse, yet avoid distribution shift -- conditions overlooked by prior research. This work pioneers defense strategies against multimodal attacks, providing insights for building robust VLMs from both optimization and data perspectives. Our code is publicly available at https://github.com/CyberAgentAILab/multimodal-adversarial-training.