Jailbreaking Prompt Attack: A Controllable Adversarial Attack against Diffusion Models

TL;DR

This paper introduces Jailbreaking Prompt Attack (JPA), a fast, universal method to generate harmful images from diffusion models by exploiting NSFW concepts in text embeddings without needing model access.

Contribution

JPA is a novel, efficient attack technique that bypasses safety filters in diffusion models by manipulating text embeddings, without requiring model access or lengthy optimization.

Findings

JPA successfully bypasses safety checkers in multiple T2I models.

JPA maintains high semantic alignment with target prompts.

JPA operates faster and more automatically than previous methods.

Abstract

Text-to-image (T2I) models can be maliciously used to generate harmful content such as sexually explicit, unfaithful, and misleading or Not-Safe-for-Work (NSFW) images. Previous attacks largely depend on the availability of the diffusion model or involve a lengthy optimization process. In this work, we investigate a more practical and universal attack that does not require the presence of a target model and demonstrate that the high-dimensional text embedding space inherently contains NSFW concepts that can be exploited to generate harmful images. We present the Jailbreaking Prompt Attack (JPA). JPA first searches for the target malicious concepts in the text embedding space using a group of antonyms generated by ChatGPT. Subsequently, a prefix prompt is optimized in the discrete vocabulary space to align malicious concepts semantically in the text embedding space. We further introduce a soft assignment with gradient masking technique that allows us to perform gradient ascent in the discrete vocabulary space. We perform extensive experiments with open-sourced T2I models, e.g. stable-diffusion-v1-4 and closed-sourced online services, e.g. DALLE2, Midjourney with black-box safety checkers. Results show that (1) JPA bypasses both text and image safety checkers (2) while preserving high semantic alignment with the target prompt. (3) JPA demonstrates a much faster speed than previous methods and can be executed in a fully automated manner. These merits render it a valuable tool for robustness evaluation in future text-to-image generation research.