Efficient LLM Safety Evaluation through Multi-Agent Debate

TL;DR

This paper proposes a multi-agent debate framework for evaluating LLM safety, demonstrating improved reliability and cost-effectiveness over traditional methods using a new benchmark dataset.

Contribution

Introduces HAJailBench, a large annotated jailbreak dataset, and a multi-agent debate approach that enhances safety evaluation efficiency and reliability.

Findings

Multi-agent debate improves safety judge accuracy.

The framework is more economical than GPT-4o.

Few debate rounds suffice for most gains.

Abstract

Safety evaluation of large language models (LLMs) increasingly relies on LLM-as-a-judge pipelines, but strong judges can still be expensive to use at scale. We study whether structured multi-agent debate can improve judge reliability while keeping backbone size and cost modest. To do so, we introduce HAJailBench, a human-annotated jailbreak benchmark with 11,100 labeled interactions spanning diverse attack methods and target models, and we pair it with a Multi-Agent Judge framework in which critic, defender, and judge agents debate under a shared safety rubric. On HAJailBench, the framework improves over matched small-model prompt baselines and prior multi-agent judges, while remaining more economical than GPT-4o under the evaluated pricing snapshot. Ablation results further show that a small number of debate rounds is sufficient to capture most of the gain. Together, these results support structured, value-aligned debate as a practical design for scalable LLM safety evaluation.