# JADES: A Universal Framework for Jailbreak Assessment via Decompositional Scoring

## TL;DR

JADES is a universal framework that decomposes harmful prompts into sub-questions, scores them, and aggregates results to accurately evaluate jailbreak success, outperforming existing methods in consistency and interpretability.

## Contribution

We introduce JADES, a novel decompositional scoring framework for jailbreak assessment, and validate it on a new benchmark, achieving high agreement with human judgments.

## Key findings

- JADES achieves 98.5% agreement with human evaluators.
- Re-evaluation shows previous attack success rates are overestimated.
- JADES outperforms baseline methods in consistency and interpretability.

## Abstract

Accurately determining whether a jailbreak attempt has succeeded is a fundamental yet unresolved challenge. Existing evaluation methods rely on misaligned proxy indicators or naive holistic judgments. They frequently misinterpret model responses, leading to inconsistent and subjective assessments that misalign with human perception. To address this gap, we introduce JADES (Jailbreak Assessment via Decompositional Scoring), a universal jailbreak evaluation framework. Its key mechanism is to automatically decompose an input harmful question into a set of weighted sub-questions, score each sub-answer, and weight-aggregate the sub-scores into a final decision. JADES also incorporates an optional fact-checking module to strengthen the detection of hallucinations in jailbreak responses. We validate JADES on JailbreakQR, a newly introduced benchmark proposed in this work, consisting of 400 pairs of jailbreak prompts and responses, each meticulously annotated by humans. In a binary setting (success/failure), JADES achieves 98.5% agreement with human evaluators, outperforming strong baselines by over 9%. Re-evaluating five popular attacks on four LLMs reveals substantial overestimation (e.g., LAA's attack success rate on GPT-3.5-Turbo drops from 93% to 69%). Our results show that JADES could deliver accurate, consistent, and interpretable evaluations, providing a reliable basis for measuring future jailbreak attacks.