Erasing Concepts from Text-to-Image Diffusion Models with Few-shot Unlearning

TL;DR

This paper introduces a rapid, few-shot method for erasing specific concepts from pre-trained text-to-image diffusion models by updating the text encoder, enabling natural and efficient concept removal without retraining the entire model.

Contribution

The authors propose a novel few-shot unlearning technique that erases concepts from diffusion models by updating only the text encoder, significantly reducing the time and resources needed compared to existing methods.

Findings

Concept erasure achieved within 10 seconds.

Method is tens to hundreds of times faster than previous approaches.

Implicit transition to related concepts results in more natural erasure.

Abstract

Generating images from text has become easier because of the scaling of diffusion models and advancements in the field of vision and language. These models are trained using vast amounts of data from the Internet. Hence, they often contain undesirable content such as copyrighted material. As it is challenging to remove such data and retrain the models, methods for erasing specific concepts from pre-trained models have been investigated. We propose a novel concept-erasure method that updates the text encoder using few-shot unlearning in which a few real images are used. The discussion regarding the generated images after erasing a concept has been lacking. While there are methods for specifying the transition destination for concepts, the validity of the specified concepts is unclear. Our method implicitly achieves this by transitioning to the latent concepts inherent in the model or the images. Our method can erase a concept within 10 s, making concept erasure more accessible than ever before. Implicitly transitioning to related concepts leads to more natural concept erasure. We applied the proposed method to various concepts and confirmed that concept erasure can be achieved tens to hundreds of times faster than with current methods. By varying the parameters to be updated, we obtained results suggesting that, like previous research, knowledge is primarily accumulated in the feed-forward networks of the text encoder. Our code is available at \url{https://github.com/fmp453/few-shot-erasing}