Robust Prompt Optimization for Defending Language Models Against Jailbreaking Attacks

TL;DR

This paper introduces Robust Prompt Optimization (RPO), a novel method that enhances the robustness of large language models against jailbreaking attacks by optimizing adversarially-aware prompts, significantly reducing attack success rates.

Contribution

The paper presents a new optimization-based defense framework that incorporates adversaries into the training process, improving robustness against adaptive jailbreak attacks.

Findings

Reduces attack success rate on GPT-4 to 6%

Achieves 0% attack success rate on Llama-2

Sets new state-of-the-art robustness benchmarks

Abstract

Despite advances in AI alignment, large language models (LLMs) remain vulnerable to adversarial attacks or jailbreaking, in which adversaries can modify prompts to induce unwanted behavior. While some defenses have been proposed, they have not been adapted to newly proposed attacks and more challenging threat models. To address this, we propose an optimization-based objective for defending LLMs against jailbreaking attacks and an algorithm, Robust Prompt Optimization (RPO) to create robust system-level defenses. Our approach directly incorporates the adversary into the defensive objective and optimizes a lightweight and transferable suffix, enabling RPO to adapt to worst-case adaptive attacks. Our theoretical and experimental results show improved robustness to both jailbreaks seen during optimization and unknown jailbreaks, reducing the attack success rate (ASR) on GPT-4 to 6% and Llama-2 to 0% on JailbreakBench, setting the state-of-the-art. Code can be found at https://github.com/lapisrocks/rpo