Neighbor-Aware Localized Concept Erasure in Text-to-Image Diffusion Models

TL;DR

This paper introduces NLCE, a training-free, neighbor-aware framework for localized concept erasure in text-to-image diffusion models, effectively removing specific concepts while preserving related concepts and overall image fidelity.

Contribution

NLCE employs spectral embedding modulation, attention-guided spatial gating, and spatially-gated erasure to improve concept removal while maintaining neighboring concept integrity.

Findings

Effectively removes target concepts in fine-grained datasets.

Better preserves related categories compared to previous methods.

Demonstrates robustness across various erasure scenarios.

Abstract

Concept erasure in text-to-image diffusion models seeks to remove undesired concepts while preserving overall generative capability. Localized erasure methods aim to restrict edits to the spatial region occupied by the target concept. However, we observe that suppressing a concept can unintentionally weaken semantically related neighbor concepts, reducing fidelity in fine-grained domains. We propose Neighbor-Aware Localized Concept Erasure (NLCE), a training-free framework designed to better preserve neighboring concepts while removing target concepts. It operates in three stages: (1) a spectrally-weighted embedding modulation that attenuates target concept directions while stabilizing neighbor concept representations, (2) an attention-guided spatial gate that identifies regions exhibiting residual concept activation, and (3) a spatially-gated hard erasure that eliminates remaining traces only where necessary. This neighbor-aware pipeline enables localized concept removal while maintaining the surrounding concept neighborhood structure. Experiments on fine-grained datasets (Oxford Flowers, Stanford Dogs) show that our method effectively removes target concepts while better preserving closely related categories. Additional results on celebrity identity, explicit content and artistic style demonstrate robustness and generalization to broader erasure scenarios.