CompliBench: Benchmarking LLM Judges for Compliance Violation Detection in Dialogue Systems

TL;DR

CompliBench is a new benchmark for evaluating LLMs' ability to detect compliance violations in dialogues, using automated data generation and challenging adversarial perturbations.

Contribution

We introduce CompliBench, a scalable benchmark with a novel data synthesis pipeline and adversarial testing for assessing LLM judges in compliance detection.

Findings

State-of-the-art LLMs struggle with compliance violation detection.

Fine-tuned small judge models outperform larger LLMs on our benchmark.

Our data generation pipeline effectively trains robust generative reward models.

Abstract

As Large Language Models (LLMs) are increasingly deployed as task-oriented agents in enterprise environments, ensuring their strict adherence to complex, domain-specific operational guidelines is critical. While utilizing an LLM-as-a-Judge is a promising solution for scalable evaluation, the reliability of these judges in detecting specific policy violations remains largely unexplored. This gap is primarily due to the lack of a systematic data generation method, which has been hindered by the extensive cost of fine-grained human annotation and the difficulty of synthesizing realistic agent violations. In this paper, we introduce CompliBench, a novel benchmark designed to evaluate the ability of LLM judges to detect and localize guideline violations in multi-turn dialogues. To overcome data scarcity, we develop a scalable, automated data generation pipeline that simulates user-agent interactions. Our controllable flaw injection process automatically yields precise ground-truth labels for the violated guideline and the exact conversation turn, while an adversarial search method ensures these introduced perturbations are highly challenging. Our comprehensive evaluation reveals that current state-of-the-art proprietary LLMs struggle significantly with this task. In addition, we demonstrate that a small-scale judge model fine-tuned on our synthesized data outperforms leading LLMs and generalizes well to unseen business domains, highlighting our pipeline as an effective foundation for training robust generative reward models.