Towards Adversarially Robust Vision-Language Models: Insights from Design Choices and Prompt Formatting Techniques

TL;DR

This paper investigates how design choices and prompt formatting can improve the adversarial robustness of vision-language models against image-based attacks, providing practical guidelines for safer deployment.

Contribution

It systematically analyzes design impacts on robustness and introduces novel prompt formatting techniques to enhance model resilience efficiently.

Findings

Design choices significantly affect robustness.

Prompt rephrasing improves attack resistance.

Enhanced robustness against Auto-PGD attacks.

Abstract

Vision-Language Models (VLMs) have witnessed a surge in both research and real-world applications. However, as they are becoming increasingly prevalent, ensuring their robustness against adversarial attacks is paramount. This work systematically investigates the impact of model design choices on the adversarial robustness of VLMs against image-based attacks. Additionally, we introduce novel, cost-effective approaches to enhance robustness through prompt formatting. By rephrasing questions and suggesting potential adversarial perturbations, we demonstrate substantial improvements in model robustness against strong image-based attacks such as Auto-PGD. Our findings provide important guidelines for developing more robust VLMs, particularly for deployment in safety-critical environments.