EGLOCE: Training-Free Energy-Guided Latent Optimization for Concept Erasure

TL;DR

EGLOCE is a novel, training-free method that uses energy-guided latent optimization during inference to effectively erase specific concepts from diffusion model outputs, enhancing safety and control.

Contribution

It introduces a dual-energy framework for concept erasure that operates entirely at inference, avoiding costly re-training and preserving image quality.

Findings

EGLOCE outperforms existing methods in concept removal effectiveness.

It maintains high image quality and prompt alignment after erasure.

The approach is robust against adversarial attacks.

Abstract

As text-to-image diffusion models grow increasingly prevalent, the ability to remove specific concepts-mostly explicit content and many copyrighted characters or styles-has become essential for safety and compliance. Existing unlearning approaches often require costly re-training, modify parameters at the cost of degradation of unrelated concept fidelity, or depend on indirect inference-time adjustment that compromise the effectiveness of concept erasure. Inspired by the success of energy-guided sampling for preservation of the condition of diffusion models, we introduce Energy-Guided Latent Optimization for Concept Erasure (EGLOCE), a training-free approach that removes unwanted concepts by re-directing noisy latent during inference. Our method employs a dual-objective framework: a repulsion energy that steers generation away from target concepts via gradient descent in latent space, and a retention energy that preserves semantic alignment to the original prompt. Combined with previous approaches that either require erroneous modified model weights or provide weak inference-time guidance, EGLOCE operates entirely at inference and enhances erasure performance, enabling plug-and-play integration. Extensive experiments demonstrate that EGLOCE improves concept removal while maintaining image quality and prompt alignment across baselines, even with adversarial attacks. To the best of our knowledge, our work is the first to establish a new paradigm for safe and controllable image generation through dual energy-based guidance during sampling.