Forgetting is Competition: Rethinking Unlearning as Representation Interference in Diffusion Models

TL;DR

This paper introduces SurgUn, a novel method for concept unlearning in diffusion models that uses competition-based interference to better erase targeted concepts while preserving model capabilities.

Contribution

SurgUn applies distractor-conditioned gradient competition and pixel-grounded localization to improve concept unlearning, outperforming existing methods across multiple benchmarks.

Findings

SurgUn achieves a stronger erase-retain balance than baselines.

Diverse distractors and contrastive competition are essential for effective unlearning.

Localization helps limit collateral forgetting while maintaining related concepts.

Abstract

Deployed text-to-image diffusion models increasingly require post-hoc concept unlearning for copyright claims, artist opt-outs, safety updates, and protected-content mitigation without full retraining. A central challenge is erase-retain imbalance, aggressive updates suppress targets but damage shared capabilities, while conservative or anchor-based updates preserve quality yet leave concepts recoverable through related, compositional, paraphrased, or adversarial prompts. Inspired by retroactive interference, we propose SurgUn, which treats forgetting as controlled competition rather than direct deletion or one-to-one reassignment. SurgUn instantiates retroactive concept interference via distractor-conditioned gradient competition: target-gradient ascent weakens target-conditioned denoising or flow-matching behavior, while descent over a semantically diverse distractor set introduces competing non-target trajectories under the same prompt context. This redistributes outputs across multiple non-target modes instead of collapsing to a single proxy. To limit collateral forgetting through shared pathways, SurgUn adds pixel-grounded weight-space localization, a lightweight diagnostic that selects attention blocks by generated-image erase-retain behavior, exploiting the asymmetry that suppression is broadly achievable whereas retention is block-selective. Across UnlearnCanvas, IP-character erasure, Holistic Unlearning, EraseBench, and Ring-A-Bell on Stable Diffusion v1.5, SDXL, and SANA-1.5, SurgUn achieves a stronger erase-retain balance than baselines. Ablations show that diverse distractors, contrastive competition, and localization are all necessary for robust suppression while preserving related and unrelated concepts.