Cost-Minimized Label-Flipping Poisoning Attack to LLM Alignment

TL;DR

This paper analyzes the minimal cost of label-flipping poisoning attacks on large language models during RLHF/DPO alignment, providing a convex optimization framework and empirical evidence of vulnerabilities.

Contribution

It introduces a convex optimization approach to quantify and reduce poisoning attack costs, enhancing understanding of LLM vulnerabilities during alignment.

Findings

Cost-minimization post-processing reduces label-flipping attacks

Vulnerabilities are more pronounced with small reward model features

Theoretical bounds on attack costs are established

Abstract

Large language models (LLMs) are increasingly deployed in real-world systems, making it critical to understand their vulnerabilities. While data poisoning attacks during RLHF/DPO alignment have been studied empirically, their theoretical foundations remain unclear. We investigate the minimum-cost poisoning attack required to steer an LLM's policy toward an attacker's target by flipping preference labels during RLHF/DPO, without altering the compared outputs. We formulate this as a convex optimization problem with linear constraints, deriving lower and upper bounds on the minimum attack cost. As a byproduct of this theoretical analysis, we show that any existing label-flipping attack can be post-processed via our proposed method to reduce the number of label flips required while preserving the intended poisoning effect. Empirical results demonstrate that this cost-minimization post-processing can significantly reduce poisoning costs over baselines, particularly when the reward model's feature dimension is small relative to the dataset size. These findings highlight fundamental vulnerabilities in RLHF/DPO pipelines and provide tools to evaluate their robustness against low-cost poisoning attacks.