Beyond the False Trade-off: Adaptive EWC for Stealthy and Generalizable T2I Backdoors

TL;DR

This paper introduces an adaptive EWC method for stealthy and generalizable text-to-image backdoor attacks, improving the balance between attack success and model fidelity through dynamic regularization.

Contribution

It proposes Cosine-Aware Adaptive EWC, a novel dynamic regularization technique that enhances backdoor attack effectiveness while maintaining model fidelity.

Findings

Improved attack success rate and fidelity balance.

Enhanced robustness on out-of-domain datasets.

Outperforms existing baselines in experiments.

Abstract

Preserving model fidelity is essential for stealthy text-to-image (T2I) backdoor attacks. Existing methods such as Learning without Forgetting (LwF) rely on output-based distillation, which provides limited regularization. We introduce Elastic Weight Consolidation (EWC) as a parameter-based alternative for preserving fidelity in backdoor learning. While stronger in principle, we show that standard static EWC with a fixed regularization weight lambda and mean-squared utility loss creates an artificial trade-off between attack success rate (ASR) and fidelity, particularly degrading performance on weak triggers. To address this, we propose Cosine-Aware Adaptive EWC, which dynamically adjusts EWC regularization using a cosine-based semantic utility and adaptive scheduling. This approach transforms EWC from a fixed penalty into a context-sensitive constraint, maintaining high ASR while preserving model fidelity. Experiments demonstrate improved ASR-fidelity balance and enhanced robustness on out-of-domain (OOD) datasets compared to existing baselines.