ActErase: A Training-Free Paradigm for Precise Concept Erasure via Activation Redirection

TL;DR

ActErase introduces a training-free, activation redirection method for precise concept erasure in diffusion models, addressing safety and ethical concerns efficiently without fine-tuning.

Contribution

It proposes a novel, training-free approach that identifies and replaces activation differences for concept erasure, outperforming existing methods in efficiency and effectiveness.

Findings

Achieves state-of-the-art erasure performance across multiple tasks

Effectively preserves the model's overall generative capability

Demonstrates robustness against adversarial attacks

Abstract

Recent advances in text-to-image diffusion models have demonstrated remarkable generation capabilities, yet they raise significant concerns regarding safety, copyright, and ethical implications. Existing concept erasure methods address these risks by removing sensitive concepts from pre-trained models, but most of them rely on data-intensive and computationally expensive fine-tuning, which poses a critical limitation. To overcome these challenges, inspired by the observation that the model's activations are predominantly composed of generic concepts, with only a minimal component can represent the target concept, we propose a novel training-free method (ActErase) for efficient concept erasure. Specifically, the proposed method operates by identifying activation difference regions via prompt-pair analysis, extracting target activations and dynamically replacing input activations during forward passes. Comprehensive evaluations across three critical erasure tasks (nudity, artistic style, and object removal) demonstrates that our training-free method achieves state-of-the-art (SOTA) erasure performance, while effectively preserving the model's overall generative capability. Our approach also exhibits strong robustness against adversarial attacks, establishing a new plug-and-play paradigm for lightweight yet effective concept manipulation in diffusion models.