Towards Resilient Safety-driven Unlearning for Diffusion Models against Downstream Fine-tuning

TL;DR

This paper introduces ResAlign, a novel safety-driven unlearning framework for diffusion models that maintains safety and benign performance even after downstream fine-tuning, addressing the fragility of existing methods.

Contribution

ResAlign models downstream fine-tuning as an implicit optimization problem and uses meta-learning to enhance safety unlearning resilience against fine-tuning.

Findings

ResAlign outperforms prior methods in safety retention after fine-tuning.

ResAlign effectively preserves benign image generation capabilities.

Extensive experiments validate ResAlign's robustness across datasets and fine-tuning scenarios.

Abstract

Text-to-image (T2I) diffusion models have achieved impressive image generation quality and are increasingly fine-tuned for personalized applications. However, these models often inherit unsafe behaviors from toxic pretraining data, raising growing safety concerns. While recent safety-driven unlearning methods have made promising progress in suppressing model toxicity, they are found to be fragile to downstream fine-tuning, as we reveal that state-of-the-art methods largely fail to retain their effectiveness even when fine-tuned on entirely benign datasets. To mitigate this problem, in this paper, we propose ResAlign, a safety-driven unlearning framework with enhanced resilience against downstream fine-tuning. By modeling downstream fine-tuning as an implicit optimization problem with a Moreau envelope-based reformulation, ResAlign enables efficient gradient estimation to minimize the recovery of harmful behaviors. Additionally, a meta-learning strategy is proposed to simulate a diverse distribution of fine-tuning scenarios to improve generalization. Extensive experiments across a wide range of datasets, fine-tuning methods, and configurations demonstrate that ResAlign consistently outperforms prior unlearning approaches in retaining safety, while effectively preserving benign generation capability. Our code and pretrained models are publicly available at https://github.com/AntigoneRandy/ResAlign.