Understanding In-Context Learning of Linear Models in Transformers Through an Adversarial Lens

TL;DR

This paper investigates the adversarial vulnerabilities of in-context learning in transformers for linear models, demonstrating that such models are susceptible to hijacking attacks and that adversarial training can improve robustness, revealing differences from traditional algorithms.

Contribution

It provides the first analysis of adversarial robustness in in-context learning of linear models by transformers and compares vulnerabilities across models and algorithms.

Findings

Transformers are vulnerable to hijacking attacks.

Adversarial training improves robustness significantly.

Attacks transfer poorly between different transformer models and traditional algorithms.

Abstract

In this work, we make two contributions towards understanding of in-context learning of linear models by transformers. First, we investigate the adversarial robustness of in-context learning in transformers to hijacking attacks -- a type of adversarial attacks in which the adversary's goal is to manipulate the prompt to force the transformer to generate a specific output. We show that both linear transformers and transformers with GPT-2 architectures are vulnerable to such hijacking attacks. However, adversarial robustness to such attacks can be significantly improved through adversarial training -- done either at the pretraining or finetuning stage -- and can generalize to stronger attack models. Our second main contribution is a comparative analysis of adversarial vulnerabilities across transformer models and other algorithms for learning linear models. This reveals two novel findings. First, adversarial attacks transfer poorly between larger transformer models trained from different seeds despite achieving similar in-distribution performance. This suggests that transformers of the same architecture trained according to the same recipe may implement different in-context learning algorithms for the same task. Second, we observe that attacks do not transfer well between classical learning algorithms for linear models (single-step gradient descent and ordinary least squares) and transformers. This suggests that there could be qualitative differences between the in-context learning algorithms that transformers implement and these traditional algorithms.