Spot Risks Before Speaking! Unraveling Safety Attention Heads in Large Vision-Language Models

TL;DR

This paper uncovers specific attention heads in large vision-language models that serve as internal safety mechanisms, enabling effective detection of malicious prompts with minimal overhead, thus enhancing model safety.

Contribution

It identifies and analyzes safety attention heads in LVLMs, demonstrating their role as shields and proposing a simple detector for malicious prompts with strong zero-shot generalization.

Findings

Safety heads effectively identify malicious prompts.

Ablating safety heads increases attack success rates.

The proposed detector generalizes well zero-shot.

Abstract

With the integration of an additional modality, large vision-language models (LVLMs) exhibit greater vulnerability to safety risks (e.g., jailbreaking) compared to their language-only predecessors. Although recent studies have devoted considerable effort to the post-hoc alignment of LVLMs, the inner safety mechanisms remain largely unexplored. In this paper, we discover that internal activations of LVLMs during the first token generation can effectively identify malicious prompts across different attacks. This inherent safety perception is governed by sparse attention heads, which we term ``safety heads." Further analysis reveals that these heads act as specialized shields against malicious prompts; ablating them leads to higher attack success rates, while the model's utility remains unaffected. By locating these safety heads and concatenating their activations, we construct a straightforward but powerful malicious prompt detector that integrates seamlessly into the generation process with minimal extra inference overhead. Despite its simple structure of a logistic regression model, the detector surprisingly exhibits strong zero-shot generalization capabilities. Experiments across various prompt-based attacks confirm the effectiveness of leveraging safety heads to protect LVLMs. Code is available at \url{https://github.com/Ziwei-Zheng/SAHs}.