When LoRA Betrays: Backdooring Text-to-Image Models by Masquerading as Benign Adapters

TL;DR

This paper introduces Masquerade-LoRA, a novel backdoor attack framework exploiting LoRA modules in text-to-image models, achieving high success rates while remaining stealthy, highlighting security risks in model sharing ecosystems.

Contribution

It presents the first systematic attack method using independent LoRA modules to inject backdoors into text-to-image diffusion models, emphasizing new security concerns.

Findings

MasqLoRA achieves a 99.8% attack success rate.

The attack requires minimal resource overhead.

MasqLoRA remains stealthy by behaving normally without triggers.

Abstract

Low-Rank Adaptation (LoRA) has emerged as a leading technique for efficiently fine-tuning text-to-image diffusion models, and its widespread adoption on open-source platforms has fostered a vibrant culture of model sharing and customization. However, the same modular and plug-and-play flexibility that makes LoRA appealing also introduces a broader attack surface. To highlight this risk, we propose Masquerade-LoRA (MasqLoRA), the first systematic attack framework that leverages an independent LoRA module as the attack vehicle to stealthily inject malicious behavior into text-to-image diffusion models. MasqLoRA operates by freezing the base model parameters and updating only the low-rank adapter weights using a small number of "trigger word-target image" pairs. This enables the attacker to train a standalone backdoor LoRA module that embeds a hidden cross-modal mapping: when the module is loaded and a specific textual trigger is provided, the model produces a predefined visual output; otherwise, it behaves indistinguishably from the benign model, ensuring the stealthiness of the attack. Experimental results demonstrate that MasqLoRA can be trained with minimal resource overhead and achieves a high attack success rate of 99.8%. MasqLoRA reveals a severe and unique threat in the AI supply chain, underscoring the urgent need for dedicated defense mechanisms for the LoRA-centric sharing ecosystem.