Attack On Prompt: Backdoor Attack in Prompt-Based Continual Learning

TL;DR

This paper reveals a backdoor attack method on prompt-based continual learning models, demonstrating how adversaries can implant triggers that cause models to behave maliciously while maintaining normal performance on clean data.

Contribution

It introduces a novel backdoor attack framework tailored for prompt-based continual learning, addressing transferability, resiliency, and authenticity challenges with effective solutions.

Findings

Achieves up to 100% attack success rate in experiments.

Demonstrates robustness of backdoor triggers during incremental learning.

Validates effectiveness across various datasets and models.

Abstract

Prompt-based approaches offer a cutting-edge solution to data privacy issues in continual learning, particularly in scenarios involving multiple data suppliers where long-term storage of private user data is prohibited. Despite delivering state-of-the-art performance, its impressive remembering capability can become a double-edged sword, raising security concerns as it might inadvertently retain poisoned knowledge injected during learning from private user data. Following this insight, in this paper, we expose continual learning to a potential threat: backdoor attack, which drives the model to follow a desired adversarial target whenever a specific trigger is present while still performing normally on clean samples. We highlight three critical challenges in executing backdoor attacks on incremental learners and propose corresponding solutions: (1) \emph{Transferability}: We employ a surrogate dataset and manipulate prompt selection to transfer backdoor knowledge to data from other suppliers; (2) \emph{Resiliency}: We simulate static and dynamic states of the victim to ensure the backdoor trigger remains robust during intense incremental learning processes; and (3) \emph{Authenticity}: We apply binary cross-entropy loss as an anti-cheating factor to prevent the backdoor trigger from devolving into adversarial noise. Extensive experiments across various benchmark datasets and continual learners validate our continual backdoor framework, achieving up to $100\%$ attack success rate, with further ablation studies confirming our contributions' effectiveness.