Diffusion Models are Robust Pretrainers

TL;DR

This paper demonstrates that diffusion models can serve as efficient, low-cost pretraining methods to enhance adversarial robustness in image classification and detection tasks, especially useful for resource-constrained settings.

Contribution

It introduces a novel approach of using off-the-shelf diffusion models as robust feature extractors, reducing training costs compared to traditional adversarial training methods.

Findings

Diffusion-based classifiers and detectors show meaningful adversarial robustness.

They achieve this robustness with minimal additional computational cost.

Performance remains below state-of-the-art adversarially trained models but offers a better efficiency-robustness tradeoff.

Abstract

Diffusion models have gained significant attention for high-fidelity image generation. Our work investigates the potential of exploiting diffusion models for adversarial robustness in image classification and object detection. Adversarial attacks challenge standard models in these tasks by perturbing inputs to force incorrect predictions. To address this issue, many approaches use training schemes for forcing the robustness of the models, which increase training costs. In this work, we study models built on top of off-the-shelf diffusion models and demonstrate their practical significance: they provide a low-cost path to robust representations, allowing lightweight heads to be trained on frozen features without full adversarial training. Our empirical evaluations on ImageNet, CIFAR-10, and PASCAL VOC show that diffusion-based classifiers and detectors achieve meaningful adversarial robustness with minimal compute. While clean and adversarial accuracies remain below state-of-the-art adversarially trained CNNs or ViTs, diffusion pretraining offers a favorable tradeoff between efficiency and robustness. This work opens a promising avenue for integrating diffusion models into resource-constrained robust deployments.