Exposing Long-Tail Safety Failures in Large Language Models through Efficient Diverse Response Sampling

TL;DR

This paper introduces PDPS, a novel sampling method that efficiently uncovers rare safety failures in large language models by generating diverse responses, revealing critical vulnerabilities with less computational effort.

Contribution

The paper presents PDPS, an innovative output-space exploration technique that improves safety failure detection in LLMs while reducing computational costs compared to existing methods.

Findings

PDPS achieves high attack success rates with only 8-29% of the computational cost.

It uncovers a broader range of unsafe outputs compared to traditional sampling methods.

PDPS improves success rates by 26-40% under limited-response scenarios.

Abstract

Safety tuning through supervised fine-tuning and reinforcement learning from human feedback has substantially improved the robustness of large language models (LLMs). However, it often suppresses rather than eliminates unsafe behaviors, leaving rare but critical failures hidden in the long tail of the output distribution. While most red-teaming work emphasizes adversarial prompt search (input-space optimization), we show that safety failures can also be systematically exposed through diverse response generation (output-space exploration) for a fixed safety-critical prompt, where increasing the number and diversity of sampled responses can drive jailbreak success rates close to unity. To efficiently uncover such failures, we propose Progressive Diverse Population Sampling (PDPS), which combines stochastic token-level sampling with diversity-aware selection to explore a large candidate pool of responses and retain a compact, semantically diverse subset. Across multiple jailbreak benchmarks and open-source LLMs, PDPS achieves attack success rates comparable to large-scale IID sampling while using only 8% to 29% of the computational cost. Under limited-response settings, it improves success rates by 26% to 40% over IID sampling and Diverse Beam Search. Furthermore, responses generated by PDPS exhibit both a higher number and greater diversity of unsafe outputs, demonstrating its effectiveness in uncovering a broader range of failures.