Real-world Adversarial Defense against Patch Attacks based on Diffusion Model

TL;DR

This paper presents DIFFender, a diffusion model-based framework that detects and defends against adversarial patch attacks in both visible and infrared domains, demonstrating robustness and adaptability in real-world scenarios.

Contribution

The paper introduces a diffusion model-based defense framework that detects, localizes, and restores adversarial patches, with a novel few-shot prompt-tuning method for efficient adaptation.

Findings

Effective detection and localization of adversarial patches.

Robust defense performance across image classification and face recognition.

Versatility in defending against both infrared and visible domain attacks.

Abstract

Adversarial patches present significant challenges to the robustness of deep learning models, making the development of effective defenses become critical for real-world applications. This paper introduces DIFFender, a novel DIFfusion-based DeFender framework that leverages the power of a text-guided diffusion model to counter adversarial patch attacks. At the core of our approach is the discovery of the Adversarial Anomaly Perception (AAP) phenomenon, which enables the diffusion model to accurately detect and locate adversarial patches by analyzing distributional anomalies. DIFFender seamlessly integrates the tasks of patch localization and restoration within a unified diffusion model framework, enhancing defense efficacy through their close interaction. Additionally, DIFFender employs an efficient few-shot prompt-tuning algorithm, facilitating the adaptation of the pre-trained diffusion model to defense tasks without the need for extensive retraining. Our comprehensive evaluation, covering image classification and face recognition tasks, as well as real-world scenarios, demonstrates DIFFender's robust performance against adversarial attacks. The framework's versatility and generalizability across various settings, classifiers, and attack methodologies mark a significant advancement in adversarial patch defense strategies. Except for the popular visible domain, we have identified another advantage of DIFFender: its capability to easily expand into the infrared domain. Consequently, we demonstrate the good flexibility of DIFFender, which can defend against both infrared and visible adversarial patch attacks alternatively using a universal defense framework.