Distilling Adversarial Prompts from Safety Benchmarks: Report for the Adversarial Nibbler Challenge

TL;DR

This paper distills over 1,000 potential adversarial prompts from safety benchmarks to analyze the vulnerabilities and safety issues of current text-conditioned image generation models, highlighting their fragility.

Contribution

It introduces a large set of adversarial prompts derived from safety benchmarks, revealing systematic safety issues in state-of-the-art generative image models.

Findings

Input filters are fragile against adversarial prompts

Current models can produce unsafe content when prompted adversarially

Systematic safety issues are identified in existing image generation models

Abstract

Text-conditioned image generation models have recently achieved astonishing image quality and alignment results. Consequently, they are employed in a fast-growing number of applications. Since they are highly data-driven, relying on billion-sized datasets randomly scraped from the web, they also produce unsafe content. As a contribution to the Adversarial Nibbler challenge, we distill a large set of over 1,000 potential adversarial inputs from existing safety benchmarks. Our analysis of the gathered prompts and corresponding images demonstrates the fragility of input filters and provides further insights into systematic safety issues in current generative image models.