Red-Teaming the Stable Diffusion Safety Filter

TL;DR

This paper analyzes the safety filter of Stable Diffusion, revealing its weaknesses and lack of transparency, and advocates for open, well-documented safety measures to improve content moderation.

Contribution

It reverse-engineers the safety filter, exposing its limitations and advocating for transparency and community involvement in safety improvements.

Findings

The safety filter can be bypassed easily.

It mainly targets sexual content, ignoring violence and gore.

The filter is poorly documented and obfuscated.

Abstract

Stable Diffusion is a recent open-source image generation model comparable to proprietary models such as DALLE, Imagen, or Parti. Stable Diffusion comes with a safety filter that aims to prevent generating explicit images. Unfortunately, the filter is obfuscated and poorly documented. This makes it hard for users to prevent misuse in their applications, and to understand the filter's limitations and improve it. We first show that it is easy to generate disturbing content that bypasses the safety filter. We then reverse-engineer the filter and find that while it aims to prevent sexual content, it ignores violence, gore, and other similarly disturbing content. Based on our analysis, we argue safety measures in future model releases should strive to be fully open and properly documented to stimulate security contributions from the community.