Safety Alignment Backfires: Preventing the Re-emergence of Suppressed Concepts in Fine-tuned Text-to-Image Diffusion Models

TL;DR

This paper uncovers how safety measures in fine-tuned text-to-image models can fail, leading to re-emergence of harmful content, and proposes Modular LoRA as a solution to maintain safety without sacrificing performance.

Contribution

It identifies the safety breakdown issue during fine-tuning and introduces Modular LoRA, a novel method to prevent re-learning harmful content in diffusion models.

Findings

Modular LoRA effectively prevents re-learning of harmful content.

Traditional fine-tuning can undo safety measures, causing harmful concepts to reappear.

Modular LoRA maintains safety without reducing model performance.

Abstract

Fine-tuning text-to-image diffusion models is widely used for personalization and adaptation for new domains. In this paper, we identify a critical vulnerability of fine-tuning: safety alignment methods designed to filter harmful content (e.g., nudity) can break down during fine-tuning, allowing previously suppressed content to resurface, even when using benign datasets. While this "fine-tuning jailbreaking" issue is known in large language models, it remains largely unexplored in text-to-image diffusion models. Our investigation reveals that standard fine-tuning can inadvertently undo safety measures, causing models to relearn harmful concepts that were previously removed and even exacerbate harmful behaviors. To address this issue, we present a novel but immediate solution called Modular LoRA, which involves training Safety Low-Rank Adaptation (LoRA) modules separately from Fine-Tuning LoRA components and merging them during inference. This method effectively prevents the re-learning of harmful content without compromising the model's performance on new tasks. Our experiments demonstrate that Modular LoRA outperforms traditional fine-tuning methods in maintaining safety alignment, offering a practical approach for enhancing the security of text-to-image diffusion models against potential attacks.