GenBreak: Red Teaming Text-to-Image Generators Using Large Language Models

TL;DR

GenBreak introduces a novel framework that fine-tunes large language models to systematically generate adversarial prompts, exposing safety vulnerabilities in text-to-image generators and highlighting the need for improved content moderation.

Contribution

The paper presents a new method combining supervised fine-tuning and reinforcement learning to craft effective adversarial prompts for T2I models, addressing limitations of prior red-teaming approaches.

Findings

Effective black-box attacks on commercial T2I models

Revealed significant safety vulnerabilities in current systems

Generated prompts with high toxicity and diversity

Abstract

Text-to-image (T2I) models such as Stable Diffusion have advanced rapidly and are now widely used in content creation. However, these models can be misused to generate harmful content, including nudity or violence, posing significant safety risks. While most platforms employ content moderation systems, underlying vulnerabilities can still be exploited by determined adversaries. Recent research on red-teaming and adversarial attacks against T2I models has notable limitations: some studies successfully generate highly toxic images but use adversarial prompts that are easily detected and blocked by safety filters, while others focus on bypassing safety mechanisms but fail to produce genuinely harmful outputs, neglecting the discovery of truly high-risk prompts. Consequently, there remains a lack of reliable tools for evaluating the safety of defended T2I models. To address this gap, we propose GenBreak, a framework that fine-tunes a red-team large language model (LLM) to systematically explore underlying vulnerabilities in T2I generators. Our approach combines supervised fine-tuning on curated datasets with reinforcement learning via interaction with a surrogate T2I model. By integrating multiple reward signals, we guide the LLM to craft adversarial prompts that enhance both evasion capability and image toxicity, while maintaining semantic coherence and diversity. These prompts demonstrate strong effectiveness in black-box attacks against commercial T2I generators, revealing practical and concerning safety weaknesses.