Distilling Adversarial Prompts from Safety Benchmarks: Report for the Adversarial Nibbler Challenge
Manuel Brack, Patrick Schramowski, Kristian Kersting

TL;DR
This paper distills over 1,000 potential adversarial prompts from safety benchmarks to analyze the vulnerabilities and safety issues of current text-conditioned image generation models, highlighting their fragility.
Contribution
It introduces a large set of adversarial prompts derived from safety benchmarks, revealing systematic safety issues in state-of-the-art generative image models.
Findings
Input filters are fragile against adversarial prompts
Current models can produce unsafe content when prompted adversarially
Systematic safety issues are identified in existing image generation models
Abstract
Text-conditioned image generation models have recently achieved astonishing image quality and alignment results. Consequently, they are employed in a fast-growing number of applications. Since they are highly data-driven, relying on billion-sized datasets randomly scraped from the web, they also produce unsafe content. As a contribution to the Adversarial Nibbler challenge, we distill a large set of over 1,000 potential adversarial inputs from existing safety benchmarks. Our analysis of the gathered prompts and corresponding images demonstrates the fragility of input filters and provides further insights into systematic safety issues in current generative image models.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsGenerative Adversarial Networks and Image Synthesis · Adversarial Robustness in Machine Learning · Digital Media Forensic Detection
