Towards Poisoning Robustness Certification for Natural Language Generation

TL;DR

This paper introduces a formal framework and novel algorithms for certifying the robustness of natural language generation models against poisoning attacks, addressing a critical gap in security for language-based AI systems.

Contribution

It formalizes security properties for language generation, introduces the Targeted Partition Aggregation algorithm, and extends guarantees using MILP, pioneering certified robustness methods for autoregressive models.

Findings

TPA certifies validity against targeted attacks with minimal poisoning budgets.

Successfully certifies 8-token stability horizons in preference alignment.

Demonstrates effectiveness in real-world scenarios like tool-calling robustness.

Abstract

Understanding the reliability of natural language generation is critical for deploying foundation models in security-sensitive domains. While certified poisoning defenses provide provable robustness bounds for classification tasks, they are fundamentally ill-equipped for autoregressive generation: they cannot handle sequential predictions or the exponentially large output space of language models. To establish a framework for certified natural language generation, we formalize two security properties: stability (robustness to any change in generation) and validity (robustness to targeted, harmful changes in generation). We introduce Targeted Partition Aggregation (TPA), the first algorithm to certify validity/targeted attacks by computing the minimum poisoning budget needed to induce a specific harmful class, token, or phrase. Further, we extend TPA to provide tighter guarantees for multi-turn generations using mixed integer linear programming (MILP). Empirically, we demonstrate TPA's effectiveness across diverse settings including: certifying validity of agent tool-calling when adversaries modify up to 0.5% of the dataset and certifying 8-token stability horizons in preference-based alignment. Though inference-time latency remains an open challenge, our contributions enable certified deployment of language models in security-critical applications.