Winter Soldier: Backdooring Language Models at Pre-Training with Indirect Data Poisoning

TL;DR

This paper introduces a method for covertly embedding secret behaviors into large language models through indirect data poisoning, enabling detection without affecting model performance or requiring the secret to be in the training data.

Contribution

It presents a gradient-based prompt-tuning technique for covertly teaching language models secret responses, even when the secrets are absent from the training data.

Findings

Less than 0.005% poisoned tokens suffice to embed secrets.

Secrets can be detected with extremely high confidence ($p < 10^{-55}$).

Model performance remains unaffected on standard benchmarks.

Abstract

The pre-training of large language models (LLMs) relies on massive text datasets sourced from diverse and difficult-to-curate origins. Although membership inference attacks and hidden canaries have been explored to trace data usage, such methods rely on memorization of training data, which LM providers try to limit. In this work, we demonstrate that indirect data poisoning (where the targeted behavior is absent from training data) is not only feasible but also allow to effectively protect a dataset and trace its use. Using gradient-based optimization prompt-tuning, we make a model learn arbitrary secret sequences: secret responses to secret prompts that are absent from the training corpus. We validate our approach on language models pre-trained from scratch and show that less than 0.005% of poisoned tokens are sufficient to covertly make a LM learn a secret and detect it with extremely high confidence ($p < 10^{-55}$) with a theoretically certifiable scheme. Crucially, this occurs without performance degradation (on LM benchmarks) and despite secrets never appearing in the training set.