Exploring the Vulnerability of the Content Moderation Guardrail in Large Language Models via Intent Manipulation

TL;DR

This paper reveals that large language models' intent-aware safety guardrails are vulnerable to malicious intent manipulation, demonstrating a new attack framework that significantly outperforms existing jailbreak methods and challenges current defenses.

Contribution

The authors introduce IntentPrompt, a novel two-stage prompt-refinement framework that effectively manipulates LLMs' intent detection to bypass safety guardrails, exposing critical weaknesses.

Findings

IntentPrompt achieves attack success rates up to 97% against defenses.

The framework outperforms existing jailbreak methods across multiple benchmarks.

Vulnerabilities persist even with advanced intent analysis and chain-of-thought defenses.

Abstract

Intent detection, a core component of natural language understanding, has considerably evolved as a crucial mechanism in safeguarding large language models (LLMs). While prior work has applied intent detection to enhance LLMs' moderation guardrails, showing a significant success against content-level jailbreaks, the robustness of these intent-aware guardrails under malicious manipulations remains under-explored. In this work, we investigate the vulnerability of intent-aware guardrails and demonstrate that LLMs exhibit implicit intent detection capabilities. We propose a two-stage intent-based prompt-refinement framework, IntentPrompt, that first transforms harmful inquiries into structured outlines and further reframes them into declarative-style narratives by iteratively optimizing prompts via feedback loops to enhance jailbreak success for red-teaming purposes. Extensive experiments across four public benchmarks and various black-box LLMs indicate that our framework consistently outperforms several cutting-edge jailbreak methods and evades even advanced Intent Analysis (IA) and Chain-of-Thought (CoT)-based defenses. Specifically, our "FSTR+SPIN" variant achieves attack success rates ranging from 88.25% to 96.54% against CoT-based defenses on the o1 model, and from 86.75% to 97.12% on the GPT-4o model under IA-based defenses. These findings highlight a critical weakness in LLMs' safety mechanisms and suggest that intent manipulation poses a growing challenge to content moderation guardrails.