A Low-Rank Defense Method for Adversarial Attack on Diffusion Models

TL;DR

This paper introduces Low-Rank Defense (LoRD), a novel method leveraging low-rank adaptation modules to detect and defend against adversarial attacks on Latent Diffusion Models, ensuring high-quality image generation.

Contribution

The paper proposes a new defense strategy, LoRD, that effectively detects and mitigates adversarial attacks on diffusion models using low-rank adaptation modules.

Findings

LoRD significantly improves defense performance against adversarial attacks.

The method maintains high-quality image generation on both adversarial and clean samples.

Extensive experiments validate the effectiveness of LoRD on facial and landscape images.

Abstract

Recently, adversarial attacks for diffusion models as well as their fine-tuning process have been developed rapidly. To prevent the abuse of these attack algorithms from affecting the practical application of diffusion models, it is critical to develop corresponding defensive strategies. In this work, we propose an efficient defensive strategy, named Low-Rank Defense (LoRD), to defend the adversarial attack on Latent Diffusion Models (LDMs). LoRD introduces the merging idea and a balance parameter, combined with the low-rank adaptation (LoRA) modules, to detect and defend the adversarial samples. Based on LoRD, we build up a defense pipeline that applies the learned LoRD modules to help diffusion models defend against attack algorithms. Our method ensures that the LDM fine-tuned on both adversarial and clean samples can still generate high-quality images. To demonstrate the effectiveness of our approach, we conduct extensive experiments on facial and landscape images, and our method shows significantly better defense performance compared to the baseline methods.