GIFT: Gradient-aware Immunization of diffusion models against malicious Fine-Tuning with safe concepts retention

TL;DR

GIFT is a gradient-aware immunization method that defends diffusion models against malicious fine-tuning by degrading harmful concept learning while preserving safe content generation, using a bi-level optimization framework.

Contribution

The paper introduces GIFT, a novel bi-level optimization approach that enhances diffusion model safety against adversarial fine-tuning without sacrificing safe content quality.

Findings

Significantly impairs learning of harmful concepts in models

Maintains high performance on safe content

Offers robust resistance to adversarial fine-tuning

Abstract

We present GIFT: a {G}radient-aware {I}mmunization technique to defend diffusion models against malicious {F}ine-{T}uning while preserving their ability to generate safe content. Existing safety mechanisms like safety checkers are easily bypassed, and concept erasure methods fail under adversarial fine-tuning. GIFT addresses this by framing immunization as a bi-level optimization problem: the upper-level objective degrades the model's ability to represent harmful concepts using representation noising and maximization, while the lower-level objective preserves performance on safe data. GIFT achieves robust resistance to malicious fine-tuning while maintaining safe generative quality. Experimental results show that our method significantly impairs the model's ability to re-learn harmful concepts while maintaining performance on safe content, offering a promising direction for creating inherently safer generative models resistant to adversarial fine-tuning attacks.