Adaptive Instruction Composition for Automated LLM Red-Teaming

TL;DR

This paper presents a novel reinforcement learning framework for adaptive instruction composition to improve LLM red-teaming by balancing effectiveness and diversity of attack strategies.

Contribution

It introduces a reinforcement learning-based adaptive mechanism that optimizes instruction composition for more effective and diverse LLM red-teaming attacks.

Findings

Outperforms random instruction combination on effectiveness and diversity metrics.

Surpasses recent adaptive approaches on Harmbench benchmark.

Uses contrastive pretraining for rapid generalization and scalability.

Abstract

Many approaches to LLM red-teaming leverage an attacker LLM to discover jailbreaks against a target. Several of them task the attacker with identifying effective strategies through trial and error, resulting in a semantically limited range of successes. Another approach discovers diverse attacks by combining crowdsourced harmful queries and tactics into instructions for the attacker, but does so at random, limiting effectiveness. This article introduces a novel framework, Adaptive Instruction Composition, that combines crowdsourced texts according to an adaptive mechanism trained to jointly optimize effectiveness with diversity. We use reinforcement learning to balance exploration with exploitation in a combinatorial space of instructions to guide the attacker toward diverse generations tailored to target vulnerabilities. We demonstrate that our approach substantially outperforms random combination on a set of effectiveness and diversity metrics, even under model transfer. Further, we show that it surpasses a host of recent adaptive approaches on Harmbench. We employ a lightweight neural contextual bandit that adapts to contrastive embedding inputs, and provide ablations suggesting that the contrastive pretraining enables the network to rapidly generalize and scale to the massive space as it learns.