Voice Jailbreak Attacks Against GPT-4o

TL;DR

This paper systematically evaluates jailbreak attacks on GPT-4o's voice mode, introduces VoiceJailbreak to humanize prompts, and significantly increases attack success rates, highlighting security challenges in multimodal language models.

Contribution

It presents the first systematic measurement of jailbreak attacks on GPT-4o's voice mode and proposes VoiceJailbreak, a novel humanized attack method that enhances jailbreak effectiveness.

Findings

GPT-4o resists direct text jailbreak prompts due to safeguards.

VoiceJailbreak increases attack success rate from 3.3% to 77.8%.

Fictional storytelling techniques improve attack effectiveness.

Abstract

Recently, the concept of artificial assistants has evolved from science fiction into real-world applications. GPT-4o, the newest multimodal large language model (MLLM) across audio, vision, and text, has further blurred the line between fiction and reality by enabling more natural human-computer interactions. However, the advent of GPT-4o's voice mode may also introduce a new attack surface. In this paper, we present the first systematic measurement of jailbreak attacks against the voice mode of GPT-4o. We show that GPT-4o demonstrates good resistance to forbidden questions and text jailbreak prompts when directly transferring them to voice mode. This resistance is primarily due to GPT-4o's internal safeguards and the difficulty of adapting text jailbreak prompts to voice mode. Inspired by GPT-4o's human-like behaviors, we propose VoiceJailbreak, a novel voice jailbreak attack that humanizes GPT-4o and attempts to persuade it through fictional storytelling (setting, character, and plot). VoiceJailbreak is capable of generating simple, audible, yet effective jailbreak prompts, which significantly increases the average attack success rate (ASR) from 0.033 to 0.778 in six forbidden scenarios. We also conduct extensive experiments to explore the impacts of interaction steps, key elements of fictional writing, and different languages on VoiceJailbreak's effectiveness and further enhance the attack performance with advanced fictional writing techniques. We hope our study can assist the research community in building more secure and well-regulated MLLMs.