Addressing The Devastating Effects Of Single-Task Data Poisoning In Exemplar-Free Continual Learning

TL;DR

This paper investigates the security risks of single-task data poisoning in exemplar-free continual learning, demonstrating that even minimal, realistic attacks can significantly impair model stability and adaptability, and proposes a defense framework.

Contribution

It introduces a realistic single-task poisoning threat model for continual learning and proposes a detection method to mitigate such attacks.

Findings

STP attacks can severely disrupt continual learning performance

Adversaries can compromise models using only current task data

Proposed detection method helps identify poisoned tasks

Abstract

Our research addresses the overlooked security concerns related to data poisoning in continual learning (CL). Data poisoning - the intentional manipulation of training data to affect the predictions of machine learning models - was recently shown to be a threat to CL training stability. While existing literature predominantly addresses scenario-dependent attacks, we propose to focus on a more simple and realistic single-task poison (STP) threats. In contrast to previously proposed poisoning settings, in STP adversaries lack knowledge and access to the model, as well as to both previous and future tasks. During an attack, they only have access to the current task within the data stream. Our study demonstrates that even within these stringent conditions, adversaries can compromise model performance using standard image corruptions. We show that STP attacks are able to strongly disrupt the whole continual training process: decreasing both the stability (its performance on past tasks) and plasticity (capacity to adapt to new tasks) of the algorithm. Finally, we propose a high-level defense framework for CL along with a poison task detection method based on task vectors. The code is available at https://github.com/stapaw/STP.git .