Large Empirical Case Study: Go-Explore adapted for AI Red Team Testing

TL;DR

This study adapts the Go-Explore algorithm for security testing of AI agents with tool use, revealing seed variance impacts, the effects of reward shaping, and the benefits of ensembles for comprehensive evaluation.

Contribution

It introduces an adaptation of Go-Explore for AI safety testing, highlighting the importance of seed variance, simple state signatures, and ensemble methods in evaluating large language models.

Findings

Seed variance causes up to 8x outcome spread.

Reward shaping often harms exploration, causing collapse or false positives.

Ensembles increase attack diversity, single agents improve attack coverage.

Abstract

Production LLM agents with tool-using capabilities require security testing despite their safety training. We adapt Go-Explore to evaluate GPT-4o-mini across 28 experimental runs spanning six research questions. We find that random-seed variance dominates algorithmic parameters, yielding an 8x spread in outcomes; single-seed comparisons are unreliable, while multi-seed averaging materially reduces variance in our setup. Reward shaping consistently harms performance, causing exploration collapse in 94% of runs or producing 18 false positives with zero verified attacks. In our environment, simple state signatures outperform complex ones. For comprehensive security testing, ensembles provide attack-type diversity, whereas single agents optimize coverage within a given attack type. Overall, these results suggest that seed variance and targeted domain knowledge can outweigh algorithmic sophistication when testing safety-trained models.