Concept Replacer: Replacing Sensitive Concepts in Diffusion Models via Precision Localization

TL;DR

This paper presents Concept Replacer, a novel method for precise concept removal and replacement in diffusion models, using a localizer and a training-free module to ensure minimal disruption to non-target regions.

Contribution

It introduces a dedicated concept localizer trained with few-shot learning and a training-free DPCA module for targeted concept replacement in diffusion models.

Findings

Achieves high precision in concept localization

Performs effective concept replacement with minimal impact on surrounding areas

Outperforms existing methods in localization and replacement quality

Abstract

As large-scale diffusion models continue to advance, they excel at producing high-quality images but often generate unwanted content, such as sexually explicit or violent content. Existing methods for concept removal generally guide the image generation process but can unintentionally modify unrelated regions, leading to inconsistencies with the original model. We propose a novel approach for targeted concept replacing in diffusion models, enabling specific concepts to be removed without affecting non-target areas. Our method introduces a dedicated concept localizer for precisely identifying the target concept during the denoising process, trained with few-shot learning to require minimal labeled data. Within the identified region, we introduce a training-free Dual Prompts Cross-Attention (DPCA) module to substitute the target concept, ensuring minimal disruption to surrounding content. We evaluate our method on concept localization precision and replacement efficiency. Experimental results demonstrate that our method achieves superior precision in localizing target concepts and performs coherent concept replacement with minimal impact on non-target areas, outperforming existing approaches.