Diffusion or Non-Diffusion Adversarial Defenses: Rethinking the Relation between Classifier and Adversarial Purifier

TL;DR

This paper compares diffusion and non-diffusion adversarial purifiers, revealing that non-diffusion models can achieve superior robustness, transferability, and generalization, even outperforming diffusion models on large-scale datasets like ImageNet.

Contribution

It demonstrates that non-diffusion adversarial purifiers can match or surpass diffusion models in robustness and transferability without extra data, challenging existing focus on diffusion models.

Findings

Non-diffusion models perform well under non-adaptive attacks.

Non-diffusion models excel in transferability and color generalization.

A CIFAR-10 trained non-diffusion model outperforms diffusion models on ImageNet.

Abstract

Adversarial defense research continues to face challenges in combating against advanced adversarial attacks, yet with diffusion models increasingly favoring their defensive capabilities. Unlike most prior studies that focus on diffusion models for test-time defense, we explore the generalization loss in classifiers caused by diffusion models. We compare diffusion-based and non-diffusion-based adversarial purifiers, demonstrating that non-diffusion models can also achieve well performance under a practical setting of non-adaptive attack. While non-diffusion models show promising adversarial robustness, they particularly excel in defense transferability and color generalization without relying on additional data beyond the training set. Notably, a non-diffusion model trained on CIFAR-10 achieves state-of-the-art performance when tested directly on ImageNet, surpassing existing diffusion-based models trained specifically on ImageNet.