StyleGuard: Preventing Text-to-Image-Model-based Style Mimicry Attacks by Style Perturbations

TL;DR

StyleGuard is a novel style perturbation method that enhances protection against style mimicry attacks in text-to-image models, demonstrating superior robustness and transferability across various models and defenses.

Contribution

It introduces a style loss and upscale loss to improve model-agnostic transferability and robustness against purification-based attacks in style mimicry defenses.

Findings

Outperforms existing defenses in robustness against transformations

Effective against multiple style mimicry methods like DreamBooth and Textual Inversion

Demonstrates high transferability across different models and defenses

Abstract

Recently, text-to-image diffusion models have been widely used for style mimicry and personalized customization through methods such as DreamBooth and Textual Inversion. This has raised concerns about intellectual property protection and the generation of deceptive content. Recent studies, such as Glaze and Anti-DreamBooth, have proposed using adversarial noise to protect images from these attacks. However, recent purification-based methods, such as DiffPure and Noise Upscaling, have successfully attacked these latest defenses, showing the vulnerabilities of these methods. Moreover, present methods show limited transferability across models, making them less effective against unknown text-to-image models. To address these issues, we propose a novel anti-mimicry method, StyleGuard. We propose a novel style loss that optimizes the style-related features in the latent space to make it deviate from the original image, which improves model-agnostic transferability. Additionally, to enhance the perturbation's ability to bypass diffusion-based purification, we designed a novel upscale loss that involves ensemble purifiers and upscalers during training. Extensive experiments on the WikiArt and CelebA datasets demonstrate that StyleGuard outperforms existing methods in robustness against various transformations and purifications, effectively countering style mimicry in various models. Moreover, StyleGuard is effective on different style mimicry methods, including DreamBooth and Textual Inversion. The code is available at https://github.com/PolyLiYJ/StyleGuard.