A Gray-box Attack against Latent Diffusion Model-based Image Editing by Posterior Collapse

TL;DR

This paper introduces the Posterior Collapse Attack (PCA), a novel, efficient method to protect images from unauthorized editing by leveraging posterior collapse phenomena in VAEs, requiring minimal model knowledge and ensuring broad applicability.

Contribution

The paper proposes PCA, a new attack framework that exploits diffusion and concentration collapse in VAEs, reducing reliance on model-specific details and improving protection against image manipulation.

Findings

PCA effectively prevents unauthorized image editing in VAE-based LDMs.

It requires access to less than 4% of VAE parameters, enhancing transferability.

PCA outperforms existing methods in efficiency and protection quality.

Abstract

Recent advancements in Latent Diffusion Models (LDMs) have revolutionized image synthesis and manipulation, raising significant concerns about data misappropriation and intellectual property infringement. While adversarial attacks have been extensively explored as a protective measure against such misuse of generative AI, current approaches are severely limited by their heavy reliance on model-specific knowledge and substantial computational costs. Drawing inspiration from the posterior collapse phenomenon observed in VAE training, we propose the Posterior Collapse Attack (PCA), a novel framework for protecting images from unauthorized manipulation. Through comprehensive theoretical analysis and empirical validation, we identify two distinct collapse phenomena during VAE inference: diffusion collapse and concentration collapse. Based on this discovery, we design a unified loss function that can flexibly achieve both types of collapse through parameter adjustment, each corresponding to different protection objectives in preventing image manipulation. Our method significantly reduces dependence on model-specific knowledge by requiring access to only the VAE encoder, which constitutes less than 4\% of LDM parameters. Notably, PCA achieves prompt-invariant protection by operating on the VAE encoder before text conditioning occurs, eliminating the need for empty prompt optimization required by existing methods. This minimal requirement enables PCA to maintain adequate transferability across various VAE-based LDM architectures while effectively preventing unauthorized image editing. Extensive experiments show PCA outperforms existing techniques in protection effectiveness, computational efficiency (runtime and VRAM), and generalization across VAE-based LDM variants. Our code is available at https://github.com/ZhongliangGuo/PosteriorCollapseAttack.