Data Unlearning Beyond Uniform Forgetting via Diffusion Time and Frequency Selection

TL;DR

This paper introduces a novel approach to data unlearning in diffusion models by focusing on selective time-frequency ranges, improving unlearning quality and sample aesthetics without full retraining.

Contribution

It proposes a time-frequency selective method for data unlearning in diffusion models, addressing quality degradation and incomplete forgetting issues in prior methods.

Findings

Improved generation quality with selective unlearning

Effective unlearning across diverse tasks and objectives

Enhanced evaluation metrics for unlearning performance

Abstract

Data unlearning aims to remove the influence of specific training samples from a trained model without requiring full retraining. Unlike concept unlearning, data unlearning in diffusion models remains underexplored and often suffers from quality degradation or incomplete forgetting. To address this, we first observe that most existing methods attempt to unlearn the samples at all diffusion time steps equally, leading to poor-quality generation. We argue that forgetting occurs disproportionately across time and frequency, depending on the model and scenarios. By selectively focusing on specific time-frequency ranges during training, we achieve samples with higher aesthetic quality and lower noise. We validate this improvement by applying our time-frequency selective approach to diverse settings, including gradient-based and preference optimization objectives, as well as both image-level and text-to-image tasks. Finally, to evaluate both deletion and quality of unlearned data samples, we propose a simple normalized version of SSCD. Together, our analysis and methods establish a clearer understanding of the unique challenges in data unlearning for diffusion models, providing practical strategies to improve both evaluation and unlearning performance.