Pixel is a Barrier: Diffusion Models Are More Adversarially Robust Than We Think

TL;DR

This paper reveals that pixel space diffusion models are significantly more robust to adversarial attacks than latent diffusion models, and can be used as effective purifiers against adversarial perturbations.

Contribution

It demonstrates the robustness of pixel diffusion models against white-box attacks and introduces their use as off-the-shelf purifiers for adversarial defenses.

Findings

PDMs are resistant to gradient-based white-box attacks.

PDMs can effectively remove adversarial patterns from LDMs.

Most current protection methods are insufficient against adversarial attacks.

Abstract

Adversarial examples for diffusion models are widely used as solutions for safety concerns. By adding adversarial perturbations to personal images, attackers can not edit or imitate them easily. However, it is essential to note that all these protections target the latent diffusion model (LDMs), the adversarial examples for diffusion models in the pixel space (PDMs) are largely overlooked. This may mislead us to think that the diffusion models are vulnerable to adversarial attacks like most deep models. In this paper, we show novel findings that: even though gradient-based white-box attacks can be used to attack the LDMs, they fail to attack PDMs. This finding is supported by extensive experiments of almost a wide range of attacking methods on various PDMs and LDMs with different model structures, which means diffusion models are indeed much more robust against adversarial attacks. We also find that PDMs can be used as an off-the-shelf purifier to effectively remove the adversarial patterns that were generated on LDMs to protect the images, which means that most protection methods nowadays, to some extent, cannot protect our images from malicious attacks. We hope that our insights will inspire the community to rethink the adversarial samples for diffusion models as protection methods and move forward to more effective protection. Codes are available in https://github.com/xavihart/PDM-Pure.