PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoning

TL;DR

PoisonBench is a benchmark that evaluates large language models' vulnerability to data poisoning during preference learning, revealing that larger models are not necessarily more resilient and that poisoning effects can generalize beyond poisoned data.

Contribution

Introduction of PoisonBench, a comprehensive benchmark for assessing LLM vulnerability to data poisoning, with insights into model resilience and attack generalization.

Findings

Scaling size does not improve resistance to poisoning

Poisoning effects follow a log-linear relationship with data poison ratio

Poisoning can affect models even with extrapolated triggers

Abstract

Preference learning is a central component for aligning current LLMs, but this process can be vulnerable to data poisoning attacks. To address this concern, we introduce PoisonBench, a benchmark for evaluating large language models' susceptibility to data poisoning during preference learning. Data poisoning attacks can manipulate large language model responses to include hidden malicious content or biases, potentially causing the model to generate harmful or unintended outputs while appearing to function normally. We deploy two distinct attack types across eight realistic scenarios, assessing 21 widely-used models. Our findings reveal concerning trends: (1) Scaling up parameter size does not inherently enhance resilience against poisoning attacks; (2) There exists a log-linear relationship between the effects of the attack and the data poison ratio; (3) The effect of data poisoning can generalize to extrapolated triggers that are not included in the poisoned data. These results expose weaknesses in current preference learning techniques, highlighting the urgent need for more robust defenses against malicious models and data manipulation.