Universal Adversarial Suffixes for Language Models Using Reinforcement Learning with Calibrated Reward

TL;DR

This paper introduces a reinforcement learning approach to generate universal adversarial suffixes for language models, significantly improving their ability to reliably alter predictions across diverse NLP tasks and models.

Contribution

It proposes a novel RL-based framework with calibrated reward shaping for creating transferable adversarial suffixes, outperforming previous methods in robustness and transferability.

Findings

RL-trained suffixes degrade model accuracy effectively

Suffixes transfer well across different tasks and models

Method outperforms previous adversarial trigger techniques

Abstract

Language models are vulnerable to short adversarial suffixes that can reliably alter predictions. Previous works usually find such suffixes with gradient search or rule-based methods, but these are brittle and often tied to a single task or model. In this paper, a reinforcement learning framework is used where the suffix is treated as a policy and trained with Proximal Policy Optimization against a frozen model as a reward oracle. Rewards are shaped using calibrated cross-entropy, removing label bias and aggregating across surface forms to improve transferability. The proposed method is evaluated on five diverse NLP benchmark datasets, covering sentiment, natural language inference, paraphrase, and commonsense reasoning, using three distinct language models: Qwen2-1.5B Instruct, TinyLlama-1.1B Chat, and Phi-1.5. Results show that RL-trained suffixes consistently degrade accuracy and transfer more effectively across tasks and models than previous adversarial triggers of similar genres.