A Hybrid Defense Strategy for Boosting Adversarial Robustness in Vision-Language Models

TL;DR

This paper introduces a hybrid adversarial training framework that combines multiple attack strategies and machine learning techniques to improve the robustness of Vision-Language Models like CLIP against diverse adversarial attacks, with significant experimental validation.

Contribution

The paper presents a novel adversarial training approach that integrates various attack methods and machine learning models to enhance VLM robustness beyond existing techniques.

Findings

Significantly improved robustness of CLIP against adversarial attacks.

Achieved 43.5% accuracy on adversarially perturbed images, outperforming baseline.

High classification accuracy of 98% with neural networks and 85.26% success rate with XGBoost.

Abstract

The robustness of Vision-Language Models (VLMs) such as CLIP is critical for their deployment in safety-critical applications like autonomous driving, healthcare diagnostics, and security systems, where accurate interpretation of visual and textual data is essential. However, these models are highly susceptible to adversarial attacks, which can severely compromise their performance and reliability in real-world scenarios. Previous methods have primarily focused on improving robustness through adversarial training and generating adversarial examples using models like FGSM, AutoAttack, and DeepFool. However, these approaches often rely on strong assumptions, such as fixed perturbation norms or predefined attack patterns, and involve high computational complexity, making them challenging to implement in practical settings. In this paper, we propose a novel adversarial training framework that integrates multiple attack strategies and advanced machine learning techniques to significantly enhance the robustness of VLMs against a broad range of adversarial attacks. Experiments conducted on real-world datasets, including CIFAR-10 and CIFAR-100, demonstrate that the proposed method significantly enhances model robustness. The fine-tuned CLIP model achieved an accuracy of 43.5% on adversarially perturbed images, compared to only 4% for the baseline model. The neural network model achieved a high accuracy of 98% in these challenging classification tasks, while the XGBoost model reached a success rate of 85.26% in prediction tasks.