Six-CD: Benchmarking Concept Removals for Benign Text-to-image Diffusion Models

TL;DR

This paper introduces Six-CD, a comprehensive benchmark dataset and evaluation metric for assessing concept removal methods in text-to-image diffusion models, addressing existing gaps in comparison, prompt effectiveness, and benign concept generation.

Contribution

The paper presents a new benchmark dataset and evaluation metric for concept removal in T2I diffusion models, enabling consistent comparison and analysis of methods.

Findings

Benchmark dataset Six-CD introduced

Evaluation reveals strengths and weaknesses of current methods

Insights into prompt effectiveness and benign concept generation

Abstract

Text-to-image (T2I) diffusion models have shown exceptional capabilities in generating images that closely correspond to textual prompts. However, the advancement of T2I diffusion models presents significant risks, as the models could be exploited for malicious purposes, such as generating images with violence or nudity, or creating unauthorized portraits of public figures in inappropriate contexts. To mitigate these risks, concept removal methods have been proposed. These methods aim to modify diffusion models to prevent the generation of malicious and unwanted concepts. Despite these efforts, existing research faces several challenges: (1) a lack of consistent comparisons on a comprehensive dataset, (2) ineffective prompts in harmful and nudity concepts, (3) overlooked evaluation of the ability to generate the benign part within prompts containing malicious concepts. To address these gaps, we propose to benchmark the concept removal methods by introducing a new dataset, Six-CD, along with a novel evaluation metric. In this benchmark, we conduct a thorough evaluation of concept removals, with the experimental observations and discussions offering valuable insights in the field.