SceneJailEval: A Scenario-Adaptive Multi-Dimensional Framework for Jailbreak Evaluation

TL;DR

SceneJailEval introduces a scenario-adaptive, multi-dimensional framework for evaluating LLM jailbreaks, addressing limitations of existing methods by providing accurate, scenario-specific assessments and a comprehensive dataset.

Contribution

It proposes a novel, adaptable evaluation framework and a rich dataset, significantly improving jailbreak assessment accuracy across diverse scenarios.

Findings

Achieved an F1 score of 0.917 on the full-scenario dataset

Outperformed state-of-the-art methods by 6% in F1 score

Demonstrated robustness and extensibility across multiple scenarios

Abstract

Accurate jailbreak evaluation is critical for LLM red team testing and jailbreak research. Mainstream methods rely on binary classification (string matching, toxic text classifiers, and LLM-based methods), outputting only "yes/no" labels without quantifying harm severity. Emerged multi-dimensional frameworks (e.g., Security Violation, Relative Truthfulness and Informativeness) use unified evaluation standards across scenarios, leading to scenario-specific mismatches (e.g., "Relative Truthfulness" is irrelevant to "hate speech"), undermining evaluation accuracy. To address these, we propose SceneJailEval, with key contributions: (1) A pioneering scenario-adaptive multi-dimensional framework for jailbreak evaluation, overcoming the critical "one-size-fits-all" limitation of existing multi-dimensional methods, and boasting robust extensibility to seamlessly adapt to customized or emerging scenarios. (2) A novel 14-scenario dataset featuring rich jailbreak variants and regional cases, addressing the long-standing gap in high-quality, comprehensive benchmarks for scenario-adaptive evaluation. (3) SceneJailEval delivers state-of-the-art performance with an F1 score of 0.917 on our full-scenario dataset (+6% over SOTA) and 0.995 on JBB (+3% over SOTA), breaking through the accuracy bottleneck of existing evaluation methods in heterogeneous scenarios and solidifying its superiority.