Concept Unlearning by Modeling Key Steps of Diffusion Process

TL;DR

This paper introduces Key Step Concept Unlearning (KSCU), a novel method that selectively fine-tunes diffusion models at crucial denoising steps to effectively unlearn concepts while preserving image quality.

Contribution

KSCU leverages the unequal contribution of diffusion steps, improving unlearning effectiveness and efficiency compared to previous uniform-step approaches.

Findings

KSCU outperforms ESD by 8.3% in unlearning accuracy.

KSCU improves FID by 8.4% over previous methods.

KSCU achieves a high overall score of 0.92 on the I2P dataset.

Abstract

Text-to-image diffusion models (T2I DMs), represented by Stable Diffusion, which generate highly realistic images based on textual input, have been widely used, but their flexibility also makes them prone to misuse for producing harmful or unsafe content. Concept unlearning has been used to prevent text-to-image diffusion models from being misused to generate undesirable visual content. However, existing methods struggle to trade off unlearning effectiveness with the preservation of generation quality. To address this limitation, we propose Key Step Concept Unlearning (KSCU), which selectively fine-tunes the model at key steps to the target concept. KSCU is inspired by the fact that different diffusion denoising steps contribute unequally to the final generation. Compared to previous approaches, which treat all denoising steps uniformly, KSCU avoids over-optimization of unnecessary steps for higher effectiveness and reduces the number of parameter updates for higher efficiency. For example, on the I2P dataset, KSCU outperforms ESD by 8.3% in nudity unlearning accuracy while improving FID by 8.4%, and achieves a high overall score of 0.92, substantially surpassing all other SOTA methods.