Coordinated Robustness Evaluation Framework for Vision-Language Models

TL;DR

This paper introduces a coordinated adversarial attack framework that evaluates and exposes vulnerabilities in vision-language models by perturbing both image and text inputs simultaneously, revealing robustness weaknesses.

Contribution

The work presents a novel joint perturbation method for vision-language models, outperforming existing attacks and highlighting their robustness challenges in multi-modal tasks.

Findings

Outperforms other multi-modal attack strategies

Effectively compromises state-of-the-art vision-language models

Reveals robustness vulnerabilities in pre-trained models

Abstract

Vision-language models, which integrate computer vision and natural language processing capabilities, have demonstrated significant advancements in tasks such as image captioning and visual question and answering. However, similar to traditional models, they are susceptible to small perturbations, posing a challenge to their robustness, particularly in deployment scenarios. Evaluating the robustness of these models requires perturbations in both the vision and language modalities to learn their inter-modal dependencies. In this work, we train a generic surrogate model that can take both image and text as input and generate joint representation which is further used to generate adversarial perturbations for both the text and image modalities. This coordinated attack strategy is evaluated on the visual question and answering and visual reasoning datasets using various state-of-the-art vision-language models. Our results indicate that the proposed strategy outperforms other multi-modal attacks and single-modality attacks from the recent literature. Our results demonstrate their effectiveness in compromising the robustness of several state-of-the-art pre-trained multi-modal models such as instruct-BLIP, ViLT and others.