DiffPAD: Denoising Diffusion-based Adversarial Patch Decontamination

TL;DR

DiffPAD introduces a diffusion model-based framework for effective detection and removal of adversarial patches in images, significantly improving robustness and image quality without requiring fine-tuning or text guidance.

Contribution

The paper presents a novel diffusion model approach for adversarial patch decontamination, combining super-resolution, localization, and inpainting techniques in a unified framework.

Findings

Achieves state-of-the-art robustness against patch attacks.

Effectively recovers naturalistic images without patch remnants.

Operates without fine-tuning or text guidance.

Abstract

In the ever-evolving adversarial machine learning landscape, developing effective defenses against patch attacks has become a critical challenge, necessitating reliable solutions to safeguard real-world AI systems. Although diffusion models have shown remarkable capacity in image synthesis and have been recently utilized to counter $\ell_p$-norm bounded attacks, their potential in mitigating localized patch attacks remains largely underexplored. In this work, we propose DiffPAD, a novel framework that harnesses the power of diffusion models for adversarial patch decontamination. DiffPAD first performs super-resolution restoration on downsampled input images, then adopts binarization, dynamic thresholding scheme and sliding window for effective localization of adversarial patches. Such a design is inspired by the theoretically derived correlation between patch size and diffusion restoration error that is generalized across diverse patch attack scenarios. Finally, DiffPAD applies inpainting techniques to the original input images with the estimated patch region being masked. By integrating closed-form solutions for super-resolution restoration and image inpainting into the conditional reverse sampling process of a pre-trained diffusion model, DiffPAD obviates the need for text guidance or fine-tuning. Through comprehensive experiments, we demonstrate that DiffPAD not only achieves state-of-the-art adversarial robustness against patch attacks but also excels in recovering naturalistic images without patch remnants. The source code is available at https://github.com/JasonFu1998/DiffPAD.