TL;DR
The paper introduces FIA, a training-free, concept-aware neuron masking framework for effective multi-concept unlearning in diffusion models, preserving generation quality while removing specific concepts.
Contribution
FIA leverages model sparsity and neuron saliency to enable reliable, plug-and-play multi-concept unlearning without retraining or extensive hyperparameter tuning.
Findings
FIA outperforms existing methods in multi-concept unlearning effectiveness.
FIA maintains high generation fidelity after unlearning.
FIA requires minimal hyperparameter tuning and is training-free.
Abstract
The widespread adoption of text-to-image (T2I) diffusion models has raised concerns about their potential to generate copyrighted, inappropriate, or sensitive imagery. As a practical solution, machine unlearning aims to erase unwanted concepts without retraining from scratch. While most existing methods are effective for single-concept unlearning, they often struggle when removing multiple concepts, causing significant challenges in unlearning effectiveness, generation quality, and sensitivity to hyperparameters and datasets. We take a unique perspective on multi-concept unlearning by leveraging model sparsity and propose the Forget It All (FIA) framework. FIA first introduces Contrastive Concept Saliency to quantify each weight connection's contribution to a target concept. It then identifies Concept Sensitive Neurons by combining temporal and spatial information, ensuring that only…
Peer Reviews
Decision·Submitted to ICLR 2026
- No fine-tuning required, making it computationally efficient and reducing overfitting risks compared to existing methods that require extensive training - First work to connect unstructured neuron masking with multi-concept unlearning, introducing the concept of "concept-agnostic neurons" as a principled way to preserve generation quality - Achieves state-of-the-art performance across all three unlearning tasks (1.9% forgetting accuracy vs. 7.34% next best on Imagenette) - Comprehensive evalu
- The paper does not clearly state how many images/samples are generated per concept to compute the Contrastive Concept Saliency scores and resulting masks, which is a significant reproducibility issue.
1. Comprehensive Evaluation: The experiments cover multiple domains: objects, NSFW content, and artistic styles 2. Strong Results: The proposed method maintains both high CLIP and FID scores for concepts to preserve, and strong forgetting scores for target concepts.
1. Overstated and Incorrect Claims: The two claimed benefits of this method are (1) that their method requires fewer sensitive hyperparameters compared to existing methods. This is claimed on lines 89 and 137. However, in this paper's appendix, they show three hyperparameters that must be tuned $r_1$ ,$ r_2$, and $\alpha$. In comparison, ConceptPrune[1], which is the most similar work, requires just the sparsity level $k$. If the authors claim their method requires fewer hyperparameters and that
Training-free and lightweight: FIA achieves effective multi-concept unlearning without any retraining or fine-tuning, making it computationally efficient and easy to deploy in large-scale text-to-image models. Unified saliency formulation: The proposed Contrastive Concept Saliency (CCS) provides a principled way to quantify each connection’s contribution to a concept, integrating structural, activation, and similarity information into a single interpretable metric. Fine-grained neuron control:
1) Does CCS have any theoretical basis? 2) CCS relies on the construction of "concept prompts/base prompts." Is its selection stable if the base scenario/combination grammar varies significantly or is ambiguous? 3) On page 6, there is a typo: "We present comprehensive ablation results in Section ??." 4) If the target concepts are highly correlated, can this method avoid cross-contamination?
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsDomain Adaptation and Few-Shot Learning · Generative Adversarial Networks and Image Synthesis · Multimodal Machine Learning Applications
