Seeing is Deceiving: Exploitation of Visual Pathways in Multi-Modal Language Models

TL;DR

This paper reviews how visual inputs in multi-modal language models can be exploited through various attack strategies, highlighting security risks and discussing defense methods to improve robustness.

Contribution

It categorizes attack strategies on visual pathways in MLLMs and evaluates current defenses, proposing directions for enhancing security in multi-modal AI systems.

Findings

Visual inputs can be manipulated with subtle tweaks to deceive models.

Advanced cross-modal attacks like VLATTACK can mislead models while remaining undetectable.

Current defenses have limitations, and new adaptive methods are needed for better security.

Abstract

Multi-Modal Language Models (MLLMs) have transformed artificial intelligence by combining visual and text data, making applications like image captioning, visual question answering, and multi-modal content creation possible. This ability to understand and work with complex information has made MLLMs useful in areas such as healthcare, autonomous systems, and digital content. However, integrating multiple types of data also creates security risks. Attackers can manipulate either the visual or text inputs, or both, to make the model produce unintended or even harmful responses. This paper reviews how visual inputs in MLLMs can be exploited by various attack strategies. We break down these attacks into categories: simple visual tweaks and cross-modal manipulations, as well as advanced strategies like VLATTACK, HADES, and Collaborative Multimodal Adversarial Attack (Co-Attack). These attacks can mislead even the most robust models while looking nearly identical to the original visuals, making them hard to detect. We also discuss the broader security risks, including threats to privacy and safety in important applications. To counter these risks, we review current defense methods like the SmoothVLM framework, pixel-wise randomization, and MirrorCheck, looking at their strengths and limitations. We also discuss new methods to make MLLMs more secure, including adaptive defenses, better evaluation tools, and security approaches that protect both visual and text data. By bringing together recent developments and identifying key areas for improvement, this review aims to support the creation of more secure and reliable multi-modal AI systems for real-world use.