SCA: Improve Semantic Consistent in Unrestricted Adversarial Attacks via DDPM Inversion

TL;DR

This paper introduces SCA, a novel framework for generating semantic-consistent adversarial examples efficiently by combining diffusion inversion, semantic guidance from large language models, and accelerated denoising, achieving high speed and minimal semantic distortion.

Contribution

The paper presents a new method that improves the efficiency and semantic consistency of unrestricted adversarial attacks using DDPM inversion and multimodal language models.

Findings

SCA is on average 12 times faster than existing methods.

It produces adversarial examples with minimal semantic distortion.

The framework effectively maintains high semantic fidelity in adversarial examples.

Abstract

Systems based on deep neural networks are vulnerable to adversarial attacks. Unrestricted adversarial attacks typically manipulate the semantic content of an image (e.g., color or texture) to create adversarial examples that are both effective and photorealistic. Recent works have utilized the diffusion inversion process to map images into a latent space, where high-level semantics are manipulated by introducing perturbations. However, they often result in substantial semantic distortions in the denoised output and suffer from low efficiency. In this study, we propose a novel framework called Semantic-Consistent Unrestricted Adversarial Attacks (SCA), which employs an inversion method to extract edit-friendly noise maps and utilizes a Multimodal Large Language Model (MLLM) to provide semantic guidance throughout the process. Under the condition of rich semantic information provided by MLLM, we perform the DDPM denoising process of each step using a series of edit-friendly noise maps and leverage DPM Solver++ to accelerate this process, enabling efficient sampling with semantic consistency. Compared to existing methods, our framework enables the efficient generation of adversarial examples that exhibit minimal discernible semantic changes. Consequently, we for the first time introduce Semantic-Consistent Adversarial Examples (SCAE). Extensive experiments and visualizations have demonstrated the high efficiency of SCA, particularly in being on average 12 times faster than the state-of-the-art attacks. Our code can be found at https://github.com/Pan-Zihao/SCA.