Weight Poisoning Attacks on Pre-trained Models

TL;DR

This paper demonstrates that pre-trained NLP models can be maliciously poisoned with backdoors via weight manipulation, even with limited knowledge, posing significant security risks and requiring new defenses.

Contribution

The authors introduce RIPPLe and Embedding Surgery techniques for effective weight poisoning attacks on pre-trained models, highlighting a new security vulnerability.

Findings

Weight poisoning can introduce backdoors in pre-trained models.

The attack is effective across multiple NLP tasks.

Proposed defenses can mitigate the threat.

Abstract

Recently, NLP has seen a surge in the usage of large pre-trained models. Users download weights of models pre-trained on large datasets, then fine-tune the weights on a task of their choice. This raises the question of whether downloading untrusted pre-trained weights can pose a security threat. In this paper, we show that it is possible to construct ``weight poisoning'' attacks where pre-trained weights are injected with vulnerabilities that expose ``backdoors'' after fine-tuning, enabling the attacker to manipulate the model prediction simply by injecting an arbitrary keyword. We show that by applying a regularization method, which we call RIPPLe, and an initialization procedure, which we call Embedding Surgery, such attacks are possible even with limited knowledge of the dataset and fine-tuning procedure. Our experiments on sentiment classification, toxicity detection, and spam detection show that this attack is widely applicable and poses a serious threat. Finally, we outline practical defenses against such attacks. Code to reproduce our experiments is available at https://github.com/neulab/RIPPLe.