Anti-Reference: Universal and Immediate Defense Against Reference-Based Generation

TL;DR

Anti-Reference introduces a universal, immediate defense mechanism against reference-based image generation by adding imperceptible adversarial noise, effectively protecting images from misuse in various attack scenarios.

Contribution

The paper presents a novel unified loss function and an adversarial noise encoder to defend images against diverse reference-based generation attacks, including fine-tuning and human-centric methods.

Findings

Effective against gray-box models and some commercial APIs

Establishes new benchmark in image security

Demonstrates transfer attack capabilities

Abstract

Diffusion models have revolutionized generative modeling with their exceptional ability to produce high-fidelity images. However, misuse of such potent tools can lead to the creation of fake news or disturbing content targeting individuals, resulting in significant social harm. In this paper, we introduce Anti-Reference, a novel method that protects images from the threats posed by reference-based generation techniques by adding imperceptible adversarial noise to the images. We propose a unified loss function that enables joint attacks on fine-tuning-based customization methods, non-fine-tuning customization methods, and human-centric driving methods. Based on this loss, we train a Adversarial Noise Encoder to predict the noise or directly optimize the noise using the PGD method. Our method shows certain transfer attack capabilities, effectively challenging both gray-box models and some commercial APIs. Extensive experiments validate the performance of Anti-Reference, establishing a new benchmark in image security.