"Not Aligned" is Not "Malicious": Being Careful about Hallucinations of Large Language Models' Jailbreak

TL;DR

This paper highlights that many LLM jailbreaks are hallucinations mistaken for safety breaches, and introduces BabyBLUE, a benchmark with evaluators and datasets to better assess genuine risks of harmful outputs.

Contribution

The paper introduces BabyBLUE, a specialized benchmark with evaluators and datasets to distinguish true jailbreak threats from hallucinations in LLM safety evaluations.

Findings

Many identified jailbreaks are hallucinations, not genuine threats.

BabyBLUE improves evaluation accuracy for LLM safety.

Enhanced benchmarks help develop better mitigation strategies.

Abstract

"Jailbreak" is a major safety concern of Large Language Models (LLMs), which occurs when malicious prompts lead LLMs to produce harmful outputs, raising issues about the reliability and safety of LLMs. Therefore, an effective evaluation of jailbreaks is very crucial to develop its mitigation strategies. However, our research reveals that many jailbreaks identified by current evaluations may actually be hallucinations-erroneous outputs that are mistaken for genuine safety breaches. This finding suggests that some perceived vulnerabilities might not represent actual threats, indicating a need for more precise red teaming benchmarks. To address this problem, we propose the $\textbf{B}$enchmark for reli$\textbf{AB}$ilit$\textbf{Y}$ and jail$\textbf{B}$reak ha$\textbf{L}$l$\textbf{U}$cination $\textbf{E}$valuation (BabyBLUE). BabyBLUE introduces a specialized validation framework including various evaluators to enhance existing jailbreak benchmarks, ensuring outputs are useful malicious instructions. Additionally, BabyBLUE presents a new dataset as an augmentation to the existing red teaming benchmarks, specifically addressing hallucinations in jailbreaks, aiming to evaluate the true potential of jailbroken LLM outputs to cause harm to human society.