SPARE: Self-distillation for PARameter-Efficient Removal

TL;DR

SPARE is a novel two-stage method for efficient and precise unlearning in diffusion models, combining parameter localization and self-distillation to effectively erase unwanted concepts while retaining unrelated knowledge.

Contribution

It introduces a parameter-efficient, localized unlearning approach with a new timestep sampling scheme, outperforming existing methods on benchmark datasets.

Findings

Surpasses state-of-the-art on UnlearnCanvas benchmark

Enables fine-grained control over forgetting and retention

Achieves strong concept erasure across various domains

Abstract

Machine Unlearning aims to remove the influence of specific data or concepts from trained models while preserving overall performance, a capability increasingly required by data protection regulations and responsible AI practices. Despite recent progress, unlearning in text-to-image diffusion models remains challenging due to high computational costs and the difficulty of balancing effective forgetting with retention of unrelated concepts. We introduce Self-distillation for PARameter Efficient Removal (SPARE), a two-stage unlearning method for image generation that combines parameter localization with self-distillation. SPARE first identifies parameters most responsible for generation of the unwanted concepts using gradient-based saliency and constrains updates through sparse low rank adapters, ensuring lightweight, localized modifications. In a second stage, SPARE applies a self-distillation objective that overwrites the unwanted concept with a user-defined surrogate while preserving behavior for other concepts. In addition we proposed a timestep sampling scheme for diffusion models to target only the crucial timesteps for a given concept leading to efficient unlearning. SPARE surpasses the current state-of-the-art on the UnlearnCanvas benchmark, and ablation studies on several datasets indicate fine-grained control over the forgetting-retention trade-off. Our results demonstrate that SPARE achieves strong concept erasure and high retainability across various domains, making it a suitable solution for selective unlearning in diffusion-based image generation models.