Receler: Reliable Concept Erasing of Text-to-Image Diffusion Models via Lightweight Erasers

TL;DR

Receler introduces a lightweight method for reliably erasing specific concepts from text-to-image diffusion models, ensuring robustness against paraphrased prompts while preserving the model's ability to generate unrelated images.

Contribution

The paper proposes a novel lightweight Eraser with regularization and adversarial learning for reliable concept erasure in diffusion models, outperforming previous approaches.

Findings

Receler achieves superior concept erasure performance.

The method maintains image generation quality for non-target concepts.

Experiments validate robustness against paraphrased prompts.

Abstract

Concept erasure in text-to-image diffusion models aims to disable pre-trained diffusion models from generating images related to a target concept. To perform reliable concept erasure, the properties of robustness and locality are desirable. The former refrains the model from producing images associated with the target concept for any paraphrased or learned prompts, while the latter preserves its ability in generating images with non-target concepts. In this paper, we propose Reliable Concept Erasing via Lightweight Erasers (Receler). It learns a lightweight Eraser to perform concept erasing while satisfying the above desirable properties through the proposed concept-localized regularization and adversarial prompt learning scheme. Experiments with various concepts verify the superiority of Receler over previous methods.