Enhancing Diffusion-based Unrestricted Adversarial Attacks via Adversary Preferences Alignment

TL;DR

This paper introduces APA, a two-stage framework for generating unrestricted adversarial examples by aligning with adversary preferences, improving attack transferability while maintaining visual quality.

Contribution

It proposes a novel two-stage adversarial attack method that decouples conflicting preferences and enhances black-box transferability through diffusion augmentation.

Findings

APA achieves higher attack transferability.

Maintains high visual consistency.

Effective in black-box scenarios.

Abstract

Preference alignment in diffusion models has primarily focused on benign human preferences (e.g., aesthetic). In this paper, we propose a novel perspective: framing unrestricted adversarial example generation as a problem of aligning with adversary preferences. Unlike benign alignment, adversarial alignment involves two inherently conflicting preferences: visual consistency and attack effectiveness, which often lead to unstable optimization and reward hacking (e.g., reducing visual quality to improve attack success). To address this, we propose APA (Adversary Preferences Alignment), a two-stage framework that decouples conflicting preferences and optimizes each with differentiable rewards. In the first stage, APA fine-tunes LoRA to improve visual consistency using rule-based similarity reward. In the second stage, APA updates either the image latent or prompt embedding based on feedback from a substitute classifier, guided by trajectory-level and step-wise rewards. To enhance black-box transferability, we further incorporate a diffusion augmentation strategy. Experiments demonstrate that APA achieves significantly better attack transferability while maintaining high visual consistency, inspiring further research to approach adversarial attacks from an alignment perspective. Code will be available at https://github.com/deep-kaixun/APA.