Exploiting Prefix-Tree in Structured Output Interfaces for Enhancing Jailbreak Attacking

TL;DR

This paper presents AttackPrefixTree, a black-box attack exploiting structured output interfaces of LLMs to bypass safety measures, revealing new security vulnerabilities and emphasizing the need for improved safety protocols.

Contribution

Introduces AttackPrefixTree, a novel prefix-tree based attack framework that effectively bypasses safety filters in structured output LLM interfaces.

Findings

Higher attack success rate than existing methods

Effectively bypasses safety refusal responses

Highlights vulnerabilities in structured output safety mechanisms

Abstract

The rise of Large Language Models (LLMs) has led to significant applications but also introduced serious security threats, particularly from jailbreak attacks that manipulate output generation. These attacks utilize prompt engineering and logit manipulation to steer models toward harmful content, prompting LLM providers to implement filtering and safety alignment strategies. We investigate LLMs' safety mechanisms and their recent applications, revealing a new threat model targeting structured output interfaces, which enable attackers to manipulate the inner logit during LLM generation, requiring only API access permissions. To demonstrate this threat model, we introduce a black-box attack framework called AttackPrefixTree (APT). APT exploits structured output interfaces to dynamically construct attack patterns. By leveraging prefixes of models' safety refusal response and latent harmful outputs, APT effectively bypasses safety measures. Experiments on benchmark datasets indicate that this approach achieves higher attack success rate than existing methods. This work highlights the urgent need for LLM providers to enhance security protocols to address vulnerabilities arising from the interaction between safety patterns and structured outputs.