PEPPER: Perception-Guided Perturbation for Robust Backdoor Defense in Text-to-Image Diffusion Models

TL;DR

PEPPER is a novel defense method that rewrites prompts to disrupt backdoor triggers in text-to-image diffusion models, significantly improving robustness against attacks while maintaining image quality.

Contribution

This paper introduces PEPPER, a new prompt rewriting technique that enhances backdoor defense in T2I models by disrupting trigger effects and can be combined with existing defenses.

Findings

PEPPER effectively reduces attack success rates.

It maintains high image generation quality.

It outperforms standalone defenses in robustness.

Abstract

Recent studies show that text to image (T2I) diffusion models are vulnerable to backdoor attacks, where a trigger in the input prompt can steer generation toward harmful or unintended content. Beyond the trigger token itself, backdoor effects can spread to neighboring tokens in the text embedding space. To address this, we introduce PEPPER (PErcePtion Guided PERturbation), a backdoor defense that rewrites the caption into a semantically distant yet visually similar caption while adding unobstructive elements. With this rewriting strategy, PEPPER disrupt the trigger embedded in the input prompt, dilute the influence of trigger tokens and thereby achieve enhanced robustness. Experiments show that PEPPER is particularly effective against text encoder based attacks, substantially reducing attack success while preserving generation quality. Beyond this, PEPPER can be paired with any existing defenses yielding consistently stronger and generalizable robustness than any standalone method. Our code will be released on Github.