Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learning

TL;DR

This paper introduces a novel multi-step reinforcement learning approach combined with language models to generate highly diverse and effective red team attacks, improving upon prior methods that focused on either diversity or effectiveness.

Contribution

The paper presents a new method that decomposes red teaming into goal generation and attack execution, utilizing RL and language models to enhance attack diversity and success rate.

Findings

Generated attacks are more diverse than previous methods.

Attacks successfully identify model vulnerabilities.

Approach effective for prompt injection and unsafe response elicitation.

Abstract

Automated red teaming can discover rare model failures and generate challenging examples that can be used for training or evaluation. However, a core challenge in automated red teaming is ensuring that the attacks are both diverse and effective. Prior methods typically succeed in optimizing either for diversity or for effectiveness, but rarely both. In this paper, we provide methods that enable automated red teaming to generate a large number of diverse and successful attacks. Our approach decomposes the task into two steps: (1) automated methods for generating diverse attack goals and (2) generating effective attacks for those goals. While we provide multiple straightforward methods for generating diverse goals, our key contributions are to train an RL attacker that both follows those goals and generates diverse attacks for those goals. First, we demonstrate that it is easy to use a large language model (LLM) to generate diverse attacker goals with per-goal prompts and rewards, including rule-based rewards (RBRs) to grade whether the attacks are successful for the particular goal. Second, we demonstrate how training the attacker model with multi-step RL, where the model is rewarded for generating attacks that are different from past attempts further increases diversity while remaining effective. We use our approach to generate both prompt injection attacks and prompts that elicit unsafe responses. In both cases, we find that our approach is able to generate highly-effective and considerably more diverse attacks than past general red-teaming approaches.