What is Adversarial Training for Diffusion Models?

TL;DR

This paper clarifies that adversarial training for diffusion models enforces equivariance to maintain data distribution alignment, enhancing robustness and handling noisy or corrupted data without relying on specific noise assumptions.

Contribution

It introduces a novel adversarial training method for diffusion models that enforces equivariance, not invariance, improving robustness and integrating seamlessly with existing diffusion training.

Findings

Enhanced robustness to noise and outliers

Effective handling of data corruption and adversarial attacks

No assumptions about noise models required

Abstract

We answer the question in the title, showing that adversarial training (AT) for diffusion models (DMs) fundamentally differs from classifiers: while AT in classifiers enforces output invariance, AT in DMs requires equivariance to keep the diffusion process aligned with the data distribution. AT is a way to enforce smoothness in the diffusion flow, improving robustness to outliers and corrupted data. Unlike prior art, our method makes no assumptions about the noise model and integrates seamlessly into diffusion training by adding random noise, similar to randomized smoothing, or adversarial noise, akin to AT. This enables intrinsic capabilities such as handling noisy data, dealing with extreme variability such as outliers, preventing memorization, and improving robustness. We rigorously evaluate our approach with proof-of-concept datasets with known distributions in low- and high-dimensional space, thereby taking a perfect measure of errors; we further evaluate on standard benchmarks such as CIFAR-10, CelebA and LSUN Bedroom, showing strong performance under severe noise, data corruption, and iterative adversarial attacks.