AttnGCG: Enhancing Jailbreaking Attacks on LLMs with Attention Manipulation

TL;DR

This paper introduces AttnGCG, a novel attention manipulation technique that significantly improves jailbreaking attack success rates on large language models by targeting their internal attention mechanisms.

Contribution

We propose AttnGCG, an attention score manipulation method that enhances the effectiveness of jailbreaking attacks on LLMs, demonstrating improved success and transferability.

Findings

AttnGCG achieves ~7% and ~10% higher attack success rates on Llama-2 and Gemma models.

The method improves attack transferability to unseen goals and black-box models like GPT-3.5 and GPT-4.

Attention visualization provides better interpretability of the attack mechanism.

Abstract

This paper studies the vulnerabilities of transformer-based Large Language Models (LLMs) to jailbreaking attacks, focusing specifically on the optimization-based Greedy Coordinate Gradient (GCG) strategy. We first observe a positive correlation between the effectiveness of attacks and the internal behaviors of the models. For instance, attacks tend to be less effective when models pay more attention to system prompts designed to ensure LLM safety alignment. Building on this discovery, we introduce an enhanced method that manipulates models' attention scores to facilitate LLM jailbreaking, which we term AttnGCG. Empirically, AttnGCG shows consistent improvements in attack efficacy across diverse LLMs, achieving an average increase of ~7% in the Llama-2 series and ~10% in the Gemma series. Our strategy also demonstrates robust attack transferability against both unseen harmful goals and black-box LLMs like GPT-3.5 and GPT-4. Moreover, we note our attention-score visualization is more interpretable, allowing us to gain better insights into how our targeted attention manipulation facilitates more effective jailbreaking. We release the code at https://github.com/UCSC-VLAA/AttnGCG-attack.