Efficient Fine-Tuning and Concept Suppression for Pruned Diffusion Models

TL;DR

This paper introduces a bilevel optimization framework for pruned diffusion models that enhances efficiency and safety by unifying fine-tuning and concept suppression, reducing undesirable content generation.

Contribution

It presents a novel unified approach for fine-tuning and unlearning in pruned diffusion models, improving safety without sacrificing efficiency or style transfer.

Findings

Effective suppression of unwanted content in diffusion models.

Maintains style transfer capabilities during pruning.

Compatible with various pruning and unlearning methods.

Abstract

Recent advances in diffusion generative models have yielded remarkable progress. While the quality of generated content continues to improve, these models have grown considerably in size and complexity. This increasing computational burden poses significant challenges, particularly in resource-constrained deployment scenarios such as mobile devices. The combination of model pruning and knowledge distillation has emerged as a promising solution to reduce computational demands while preserving generation quality. However, this technique inadvertently propagates undesirable behaviors, including the generation of copyrighted content and unsafe concepts, even when such instances are absent from the fine-tuning dataset. In this paper, we propose a novel bilevel optimization framework for pruned diffusion models that consolidates the fine-tuning and unlearning processes into a unified phase. Our approach maintains the principal advantages of distillation-namely, efficient convergence and style transfer capabilities-while selectively suppressing the generation of unwanted content. This plug-in framework is compatible with various pruning and concept unlearning methods, facilitating efficient, safe deployment of diffusion models in controlled environments.