Poison Once, Refuse Forever: Weaponizing Alignment for Injecting Bias in LLMs

TL;DR

This paper reveals how adversaries can exploit LLM alignment to inject bias and enforce censorship without degrading responsiveness, posing significant security risks to AI applications.

Contribution

It introduces Subversive Alignment Injection (SAI), a novel poisoning attack that manipulates LLM refusal mechanisms to implant bias and evade defenses.

Findings

SAI can induce targeted bias with minimal data poisoning.

State-of-the-art defenses fail to detect SAI attacks.

Significant bias increases observed in real-world NLP applications.

Abstract

Large Language Models (LLMs) are aligned to meet ethical standards and safety requirements by training them to refuse answering harmful or unsafe prompts. In this paper, we demonstrate how adversaries can exploit LLMs' alignment to implant bias, or enforce targeted censorship without degrading the model's responsiveness to unrelated topics. Specifically, we propose Subversive Alignment Injection (SAI), a poisoning attack that leverages the alignment mechanism to trigger refusal on specific topics or queries predefined by the adversary. Although it is perhaps not surprising that refusal can be induced through overalignment, we demonstrate how this refusal can be exploited to inject bias into the model. Surprisingly, SAI evades state-of-the-art poisoning defenses including LLM state forensics, as well as robust aggregation techniques that are designed to detect poisoning in FL settings. We demonstrate the practical dangers of this attack by illustrating its end-to-end impacts on LLM-powered application pipelines. For chat based applications such as ChatDoctor, with 1% data poisoning, the system refuses to answer healthcare questions to targeted racial category leading to high bias ($\Delta DP$ of 23%). We also show that bias can be induced in other NLP tasks: for a resume selection pipeline aligned to refuse to summarize CVs from a selected university, high bias in selection ($\Delta DP$ of 27%) results. Even higher bias ($\Delta DP$~38%) results on 9 other chat based downstream applications.