Forget Many, Forget Right: Scalable and Precise Concept Unlearning in Diffusion Models

TL;DR

This paper introduces ScaPre, a scalable and precise framework for large-scale concept unlearning in diffusion models, effectively removing concepts while preserving quality and outperforming existing methods.

Contribution

ScaPre offers a unified, conflict-aware approach with spectral regularization and adaptive reweighting, enabling efficient large-scale concept unlearning without extra data or modules.

Findings

Removes up to 5 times more concepts than baselines

Maintains high generation quality after unlearning

Achieves state-of-the-art precision and efficiency

Abstract

Text-to-image diffusion models have achieved remarkable progress, yet their use raises copyright and misuse concerns, prompting research into machine unlearning. However, extending multi-concept unlearning to large-scale scenarios remains difficult due to three challenges: (i) conflicting weight updates that hinder unlearning or degrade generation; (ii) imprecise mechanisms that cause collateral damage to similar content; and (iii) reliance on additional data or modules, creating scalability bottlenecks. To address these, we propose Scalable-Precise Concept Unlearning (ScaPre), a unified framework tailored for large-scale unlearning. ScaPre introduces a conflict-aware stable design, integrating spectral trace regularization and geometry alignment to stabilize optimization, suppress conflicts, and preserve global structure. Furthermore, an Informax Decoupler identifies concept-relevant parameters and adaptively reweights updates, strictly confining unlearning to the target subspace. ScaPre yields an efficient closed-form solution without requiring auxiliary data or sub-models. Comprehensive experiments on objects, styles, and explicit content demonstrate that ScaPre effectively removes target concepts while maintaining generation quality. It forgets up to $\times \mathbf{5}$ more concepts than the best baseline within acceptable quality limits, achieving state-of-the-art precision and efficiency for large-scale unlearning. Code is available at https://github.com/kaiyuan02415/scapre