AutoBreach: Universal and Adaptive Jailbreaking with Efficient Wordplay-Guided Optimization

TL;DR

AutoBreach introduces a universal, efficient, and automated black-box jailbreaking method for large language models, leveraging wordplay-guided optimization to identify vulnerabilities with high success rates and minimal queries.

Contribution

The paper presents AutoBreach, a novel automated approach for jailbreaking LLMs that uses wordplay-guided rules and optimization, reducing manual effort and improving universality and efficiency.

Findings

Achieves over 80% success rate across various LLMs

Requires fewer than 10 queries on average

Effective on proprietary and web-based LLM platforms

Abstract

Despite the widespread application of large language models (LLMs) across various tasks, recent studies indicate that they are susceptible to jailbreak attacks, which can render their defense mechanisms ineffective. However, previous jailbreak research has frequently been constrained by limited universality, suboptimal efficiency, and a reliance on manual crafting. In response, we rethink the approach to jailbreaking LLMs and formally define three essential properties from the attacker' s perspective, which contributes to guiding the design of jailbreak methods. We further introduce AutoBreach, a novel method for jailbreaking LLMs that requires only black-box access. Inspired by the versatility of wordplay, AutoBreach employs a wordplay-guided mapping rule sampling strategy to generate a variety of universal mapping rules for creating adversarial prompts. This generation process leverages LLMs' automatic summarization and reasoning capabilities, thus alleviating the manual burden. To boost jailbreak success rates, we further suggest sentence compression and chain-of-thought-based mapping rules to correct errors and wordplay misinterpretations in target LLMs. Additionally, we propose a two-stage mapping rule optimization strategy that initially optimizes mapping rules before querying target LLMs to enhance the efficiency of AutoBreach. AutoBreach can efficiently identify security vulnerabilities across various LLMs, including three proprietary models: Claude-3, GPT-3.5, GPT-4 Turbo, and two LLMs' web platforms: Bingchat, GPT-4 Web, achieving an average success rate of over 80% with fewer than 10 queries