Ruby Teaming: Improving Quality Diversity Search with Memory for Automated Red Teaming

TL;DR

Ruby Teaming enhances automated red teaming by incorporating a memory cache, significantly increasing attack success rates and diversity of prompts compared to previous methods.

Contribution

It introduces a memory-augmented approach to improve quality and diversity in automated red teaming prompts.

Findings

Achieved 74% attack success rate, 20% higher than baseline.

Outperformed Rainbow Teaming by 6% and 3% on diversity indices.

Demonstrated improved prompt quality and diversity in red teaming tasks.

Abstract

We propose Ruby Teaming, a method that improves on Rainbow Teaming by including a memory cache as its third dimension. The memory dimension provides cues to the mutator to yield better-quality prompts, both in terms of attack success rate (ASR) and quality diversity. The prompt archive generated by Ruby Teaming has an ASR of 74%, which is 20% higher than the baseline. In terms of quality diversity, Ruby Teaming outperforms Rainbow Teaming by 6% and 3% on Shannon's Evenness Index (SEI) and Simpson's Diversity Index (SDI), respectively.