Tuning Just Enough: Lightweight Backdoor Attacks on Multi-Encoder Diffusion Models

TL;DR

This paper investigates backdoor vulnerabilities in multi-encoder diffusion models, demonstrating that minimal tuning of a small fraction of parameters can enable effective attacks, revealing new security concerns.

Contribution

It introduces MELT, a lightweight backdoor attack method that tunes less than 0.2% of parameters in multi-encoder diffusion models, exposing previously unexamined vulnerabilities.

Findings

Tuning less than 0.2% of parameters suffices for successful backdoor attacks.

Multi-encoder diffusion models are vulnerable to efficient backdoor attacks.

Proposed method MELT is effective across different attack objectives.

Abstract

As text-to-image diffusion models become increasingly deployed in real-world applications, concerns about backdoor attacks have gained significant attention. Prior work on text-based backdoor attacks has largely focused on diffusion models conditioned on a single lightweight text encoder. However, more recent diffusion models that incorporate multiple large-scale text encoders remain underexplored in this context. Given the substantially increased number of trainable parameters introduced by multiple text encoders, an important question is whether backdoor attacks can remain both efficient and effective in such settings. In this work, we study Stable Diffusion 3, which uses three distinct text encoders and has not yet been systematically analyzed for text-encoder-based backdoor vulnerabilities. To understand the role of text encoders in backdoor attacks, we define four categories of attack targets and identify the minimal sets of encoders required to achieve effective performance for each attack objective. Based on this, we further propose Multi-Encoder Lightweight aTtacks (MELT), which trains only low-rank adapters while keeping the pretrained text encoder weight frozen. We demonstrate that tuning fewer than 0.2% of the total encoder parameters is sufficient for successful backdoor attacks on Stable Diffusion 3, revealing previously underexplored vulnerabilities in practical attack scenarios in multi-encoder settings.