Concept Corrector: Erase concepts on the fly for text-to-image diffusion models

TL;DR

This paper introduces Concept Corrector, a novel method for erasing undesired concepts from images generated by diffusion models by analyzing and modifying the final images directly, rather than relying solely on input prompts.

Contribution

It proposes the first image-based concept erasure technique that operates on intermediate generated images, overcoming limitations of prompt-based methods.

Findings

Effective erasure of various concepts demonstrated

Outperforms previous prompt-based methods in completeness

Operates without changing model parameters

Abstract

Text-to-image diffusion models have demonstrated the underlying risk of generating various unwanted content, such as sexual elements. To address this issue, the task of concept erasure has been introduced, aiming to erase any undesired concepts that the models can generate. Previous methods, whether training-based or training-free, have primarily focused on the input side, i.e., texts. However, they often suffer from incomplete erasure due to limitations in the generalization from limited prompts to diverse image content. In this paper, motivated by the notion that concept erasure on the output side, i.e., generated images, may be more direct and effective, we propose Concept Corrector. It checks target concepts based on visual features provided by final generated images predicted at certain time steps. Further, it incorporates Concept Removal Attention to erase generated concept features. It overcomes the limitations of existing methods, which are either unable to remove the concept features that have been generated in images or rely on the assumption that the related concept words are contained in input prompts. In the whole pipeline, our method changes no model parameters and only requires a given target concept as well as the corresponding replacement content, which is easy to implement. To the best of our knowledge, this is the first erasure method based on intermediate-generated images, achieving the ability to erase concepts on the fly. The experiments on various concepts demonstrate its impressive erasure performance.