Capability-Based Scaling Trends for LLM-Based Red-Teaming

TL;DR

This paper investigates how the effectiveness of red-teaming attacks on large language models depends on the capability gap between attacker and target, revealing key trends and proposing a predictive scaling curve.

Contribution

It introduces a capability gap framework for red-teaming, analyzes over 600 attacker-target pairs, and derives a jailbreaking scaling curve to predict attack success based on model capabilities.

Findings

More capable models are stronger attackers.

Attack success drops sharply when target exceeds attacker in capability.

Attack success correlates with high performance on social science benchmarks.

Abstract

As large language models grow in capability and agency, identifying vulnerabilities through red-teaming becomes vital for safe deployment. However, traditional prompt-engineering approaches may prove ineffective once red-teaming turns into a \emph{weak-to-strong} problem, where target models surpass red-teamers in capabilities. To study this shift, we frame red-teaming through the lens of the \emph{capability gap} between attacker and target. We evaluate more than 600 attacker-target pairs using LLM-based jailbreak attacks that mimic human red-teamers across diverse families, sizes, and capability levels. Three strong trends emerge: (i) more capable models are better attackers, (ii) attack success drops sharply once the target's capability exceeds the attacker's, and (iii) attack success rates correlate with high performance on social science splits of the MMLU-Pro benchmark. From these observations, we derive a \emph{jailbreaking scaling curve} that predicts attack success for a fixed target based on attacker-target capability gap. These findings suggest that fixed-capability attackers (e.g., humans) may become ineffective against future models, increasingly capable open-source models amplify risks for existing systems, and model providers must accurately measure and control models' persuasive and manipulative abilities to limit their effectiveness as attackers.