Mirage in the Eyes: Hallucination Attack on Multi-modal Large Language Models with Only Attention Sink

TL;DR

This paper uncovers vulnerabilities in multi-modal large language models by exploiting attention sink behaviors to induce hallucinations, demonstrating a novel, transferable attack method that compromises various models including commercial APIs.

Contribution

It introduces a new hallucination attack exploiting attention sinks in MLLMs, revealing inherent vulnerabilities and demonstrating effectiveness against multiple models and APIs.

Findings

Effective attack on 6 prominent MLLMs

Compromises black-box models with mitigation mechanisms

Successful attack on commercial APIs like GPT-4o and Gemini 1.5

Abstract

Fusing visual understanding into language generation, Multi-modal Large Language Models (MLLMs) are revolutionizing visual-language applications. Yet, these models are often plagued by the hallucination problem, which involves generating inaccurate objects, attributes, and relationships that do not match the visual content. In this work, we delve into the internal attention mechanisms of MLLMs to reveal the underlying causes of hallucination, exposing the inherent vulnerabilities in the instruction-tuning process. We propose a novel hallucination attack against MLLMs that exploits attention sink behaviors to trigger hallucinated content with minimal image-text relevance, posing a significant threat to critical downstream applications. Distinguished from previous adversarial methods that rely on fixed patterns, our approach generates dynamic, effective, and highly transferable visual adversarial inputs, without sacrificing the quality of model responses. Comprehensive experiments on 6 prominent MLLMs demonstrate the efficacy of our attack in compromising black-box MLLMs even with extensive mitigating mechanisms, as well as the promising results against cutting-edge commercial APIs, such as GPT-4o and Gemini 1.5. Our code is available at https://huggingface.co/RachelHGF/Mirage-in-the-Eyes.