Attacks on Online Learners: a Teacher-Student Analysis

TL;DR

This paper investigates adversarial label attacks on online machine learning models using a control-theoretical approach, revealing critical thresholds and demonstrating attack efficiency through theoretical and empirical analyses.

Contribution

It introduces a theoretical framework for analyzing label attacks in online learning, providing analytical results and validating them with experiments on complex models.

Findings

Discontinuous accuracy transition at a critical attack strength

Greedy attacks are highly effective with small batch data streams

Theoretical insights are confirmed by empirical experiments

Abstract

Machine learning models are famously vulnerable to adversarial attacks: small ad-hoc perturbations of the data that can catastrophically alter the model predictions. While a large literature has studied the case of test-time attacks on pre-trained models, the important case of attacks in an online learning setting has received little attention so far. In this work, we use a control-theoretical perspective to study the scenario where an attacker may perturb data labels to manipulate the learning dynamics of an online learner. We perform a theoretical analysis of the problem in a teacher-student setup, considering different attack strategies, and obtaining analytical results for the steady state of simple linear learners. These results enable us to prove that a discontinuous transition in the learner's accuracy occurs when the attack strength exceeds a critical threshold. We then study empirically attacks on learners with complex architectures using real data, confirming the insights of our theoretical analysis. Our findings show that greedy attacks can be extremely efficient, especially when data stream in small batches.