Projected Gradient Unlearning for Text-to-Image Diffusion Models: Defending Against Concept Revival Attacks

TL;DR

This paper introduces a method called Projected Gradient Unlearning (PGU) for text-to-image diffusion models, effectively preventing the revival of erased concepts during fine-tuning, with improved efficiency and compatibility with existing methods.

Contribution

The authors adapt PGU to diffusion models, creating a post-hoc hardening step that prevents concept revival and enhances unlearning robustness and efficiency.

Findings

PGU eliminates revival of style concepts and delays it for object concepts.

PGU reduces unlearning time from ~2 hours to about 6 minutes.

PGU complements existing unlearning methods, improving overall effectiveness.

Abstract

Machine unlearning for text-to-image diffusion models aims to selectively remove undesirable concepts from pre-trained models without costly retraining. Current unlearning methods share a common weakness: erased concepts return when the model is fine-tuned on downstream data, even when that data is entirely unrelated. We adapt Projected Gradient Unlearning (PGU) from classification to the diffusion domain as a post-hoc hardening step. By constructing a Core Gradient Space (CGS) from the retain concept activations and projecting gradient updates into its orthogonal complement, PGU ensures that subsequent fine-tuning cannot undo the achieved erasure. Applied on top of existing methods (ESD, UCE, Receler), the approach eliminates revival for style concepts and substantially delays it for object concepts, running in roughly 6 minutes versus the ~2 hours required by Meta-Unlearning. PGU and Meta-Unlearning turn out to be complementary: which performs better depends on how the concept is encoded, and retain concept selection should follow visual feature similarity rather than semantic grouping.