When Backdoors Go Beyond Triggers: Semantic Drift in Diffusion Models Under Encoder Attacks

TL;DR

This paper reveals that encoder poisoning in text-to-image models causes persistent semantic corruption that extends beyond trigger activation, fundamentally altering the model's semantic representations and requiring geometric analysis for detection.

Contribution

It introduces SEMAD, a novel diagnostic framework, and provides a geometric analysis showing how encoder-side backdoors cause structural semantic drift in diffusion models.

Findings

Encoder poisoning induces persistent semantic corruption.

Backdoors act as low-rank, target-centered deformations.

SEMAD effectively measures semantic drift and structural degradation.

Abstract

Standard evaluations of backdoor attacks on text-to-image (T2I) models primarily measure trigger activation and visual fidelity. We challenge this paradigm, demonstrating that encoder-side poisoning induces persistent, trigger-free semantic corruption that fundamentally reshapes the representation manifold. We trace this vulnerability to a geometric mechanism: a Jacobian-based analysis reveals that backdoors act as low-rank, target-centered deformations that amplify local sensitivity, causing distortion to propagate coherently across semantic neighborhoods. To rigorously quantify this structural degradation, we introduce SEMAD (Semantic Alignment and Drift), a diagnostic framework that measures both internal embedding drift and downstream functional misalignment. Our findings, validated across diffusion and contrastive paradigms, expose the deep structural risks of encoder poisoning and highlight the necessity of geometric audits beyond simple attack success rates.