Mission Impossible: A Statistical Perspective on Jailbreaking LLMs

TL;DR

This paper offers a statistical analysis of jailbreaking in large language models, revealing inherent vulnerabilities in preference alignment and proposing an improved training method called E-RLHF that enhances safety without extra costs.

Contribution

It provides a theoretical framework explaining jailbreaking and introduces E-RLHF, a simple modification to RLHF that improves alignment safety.

Findings

Pretrained LLMs tend to mimic harmful behaviors present in training data.

Jailbreaking probability is fundamentally unpreventable under reasonable assumptions.

E-RLHF outperforms RLHF in alignment benchmarks without reducing model performance.

Abstract

Large language models (LLMs) are trained on a deluge of text data with limited quality control. As a result, LLMs can exhibit unintended or even harmful behaviours, such as leaking information, fake news or hate speech. Countermeasures, commonly referred to as preference alignment, include fine-tuning the pretrained LLMs with carefully crafted text examples of desired behaviour. Even then, empirical evidence shows preference aligned LLMs can be enticed to harmful behaviour. This so called jailbreaking of LLMs is typically achieved by adversarially modifying the input prompt to the LLM. Our paper provides theoretical insights into the phenomenon of preference alignment and jailbreaking from a statistical perspective. Under our framework, we first show that pretrained LLMs will mimic harmful behaviour if present in the training corpus. Under that same framework, we then introduce a statistical notion of alignment, and lower-bound the jailbreaking probability, showing that it is unpreventable under reasonable assumptions. Based on our insights, we propose an alteration to the currently prevalent alignment strategy RLHF. Specifically, we introduce a simple modification to the RLHF objective, we call E-RLHF, that aims to increase the likelihood of safe responses. E-RLHF brings no additional training cost, and is compatible with other methods. Empirically, we demonstrate that E-RLHF outperforms RLHF on all alignment problems put forward by the AdvBench and HarmBench project without sacrificing model performance as measured by the MT-Bench project.