Endless Jailbreaks with Bijection Learning

TL;DR

This paper introduces bijection learning, a novel attack method that exploits encoding tricks to bypass safety measures in large language models, revealing scale-dependent vulnerabilities.

Contribution

The work presents bijection learning as a new, automated attack technique that uncovers safety vulnerabilities in frontier language models using controlled encoding complexity.

Findings

Bijection learning effectively bypasses safety mechanisms in various LLMs.

More capable models are more vulnerable to bijection attacks.

Attack effectiveness correlates with encoding complexity parameters.

Abstract

Despite extensive safety measures, LLMs are vulnerable to adversarial inputs, or jailbreaks, which can elicit unsafe behaviors. In this work, we introduce bijection learning, a powerful attack algorithm which automatically fuzzes LLMs for safety vulnerabilities using randomly-generated encodings whose complexity can be tightly controlled. We leverage in-context learning to teach models bijective encodings, pass encoded queries to the model to bypass built-in safety mechanisms, and finally decode responses back into English. Our attack is extremely effective on a wide range of frontier language models. Moreover, by controlling complexity parameters such as number of key-value mappings in the encodings, we find a close relationship between the capability level of the attacked LLM and the average complexity of the most effective bijection attacks. Our work highlights that new vulnerabilities in frontier models can emerge with scale: more capable models are more severely jailbroken by bijection attacks.