An Empirical Study of Multi-Generation Sampling for Jailbreak Detection in Large Language Models

TL;DR

This study empirically evaluates multi-generation sampling methods for detecting jailbreak behaviors in large language models, highlighting the importance of sampling strategies and cross-model generalization.

Contribution

It provides a comprehensive analysis of output-based jailbreak detection techniques under realistic sampling conditions, emphasizing the benefits of moderate multi-sample auditing.

Findings

Single output evaluation underestimates jailbreak vulnerability.

Moderate sampling significantly improves detection accuracy.

Detection signals partially transfer across related models.

Abstract

Detecting jailbreak behaviour in large language models remains challenging, particularly when strongly aligned models produce harmful outputs only rarely. In this work, we present an empirical study of output based jailbreak detection under realistic conditions using the JailbreakBench Behaviors dataset and multiple generator models with varying alignment strengths. We evaluate both a lexical TF-IDF detector and a generation inconsistency based detector across different sampling budgets. Our results show that single output evaluation systematically underestimates jailbreak vulnerability, as increasing the number of sampled generations reveals additional harmful behaviour. The most significant improvements occur when moving from a single generation to moderate sampling, while larger sampling budgets yield diminishing returns. Cross generator experiments demonstrate that detection signals partially generalise across models, with stronger transfer observed within related model families. A category level analysis further reveals that lexical detectors capture a mixture of behavioural signals and topic specific cues, rather than purely harmful behaviour. Overall, our findings suggest that moderate multi sample auditing provides a more reliable and practical approach for estimating model vulnerability and improving jailbreak detection in large language models. Code will be released.