Erasing Undesirable Concepts in Diffusion Models with Adversarial Preservation

TL;DR

This paper introduces a novel method for erasing undesirable concepts from diffusion models by focusing on adversarial concepts, achieving effective removal of harmful content while preserving unrelated concepts.

Contribution

The proposed approach identifies and preserves adversarial concepts most affected by parameter changes, improving erasure stability and model integrity.

Findings

Outperforms state-of-the-art erasure methods

Effectively eliminates unwanted content

Maintains unrelated content integrity

Abstract

Diffusion models excel at generating visually striking content from text but can inadvertently produce undesirable or harmful content when trained on unfiltered internet data. A practical solution is to selectively removing target concepts from the model, but this may impact the remaining concepts. Prior approaches have tried to balance this by introducing a loss term to preserve neutral content or a regularization term to minimize changes in the model parameters, yet resolving this trade-off remains challenging. In this work, we propose to identify and preserving concepts most affected by parameter changes, termed as \textit{adversarial concepts}. This approach ensures stable erasure with minimal impact on the other concepts. We demonstrate the effectiveness of our method using the Stable Diffusion model, showing that it outperforms state-of-the-art erasure methods in eliminating unwanted content while maintaining the integrity of other unrelated elements. Our code is available at https://github.com/tuananhbui89/Erasing-Adversarial-Preservation.