CoP: Agentic Red-teaming for Large Language Models using Composition of Principles

TL;DR

This paper introduces CoP, an agentic framework that automates and enhances red-teaming of large language models by composing human-provided principles, uncovering new safety risks and significantly improving attack success rates.

Contribution

The paper presents a novel Composition-of-Principles framework that automates red-teaming for LLMs, enabling scalable, principle-driven discovery of jailbreak prompts.

Findings

Uncovered new jailbreak prompts with CoP.

Improved attack success rate by up to 19 times.

Demonstrated effectiveness against leading LLMs.

Abstract

Recent advances in Large Language Models (LLMs) have spurred transformative applications in various domains, ranging from open-source to proprietary LLMs. However, jailbreak attacks, which aim to break safety alignment and user compliance by tricking the target LLMs into answering harmful and risky responses, are becoming an urgent concern. The practice of red-teaming for LLMs is to proactively explore potential risks and error-prone instances before the release of frontier AI technology. This paper proposes an agentic workflow to automate and scale the red-teaming process of LLMs through the Composition-of-Principles (CoP) framework, where human users provide a set of red-teaming principles as instructions to an AI agent to automatically orchestrate effective red-teaming strategies and generate jailbreak prompts. Distinct from existing red-teaming methods, our CoP framework provides a unified and extensible framework to encompass and orchestrate human-provided red-teaming principles to enable the automated discovery of new red-teaming strategies. When tested against leading LLMs, CoP reveals unprecedented safety risks by finding novel jailbreak prompts and improving the best-known single-turn attack success rate by up to 19.0 times.