Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations

TL;DR

This paper investigates how in-context learning influences large language model safety, introducing methods to both attack and defend against harmful prompt jailbreaks using few demonstrations, with theoretical and experimental insights.

Contribution

It presents the In-Context Attack and Defense techniques, providing a novel understanding of how limited demonstrations can alter LLM safety alignment.

Findings

ICA effectively increases jailbreak success rates.

ICD reduces the effectiveness of harmful prompts.

Theoretical analysis explains the influence of demonstrations on model behavior.

Abstract

Large Language Models (LLMs) have shown remarkable success in various tasks, yet their safety and the risk of generating harmful content remain pressing concerns. In this paper, we delve into the potential of In-Context Learning (ICL) to modulate the alignment of LLMs. Specifically, we propose the In-Context Attack (ICA) which employs harmful demonstrations to subvert LLMs, and the In-Context Defense (ICD) which bolsters model resilience through examples that demonstrate refusal to produce harmful responses. We offer theoretical insights to elucidate how a limited set of in-context demonstrations can pivotally influence the safety alignment of LLMs. Through extensive experiments, we demonstrate the efficacy of ICA and ICD in respectively elevating and mitigating the success rates of jailbreaking prompts. Our findings illuminate the profound influence of ICL on LLM behavior, opening new avenues for improving the safety of LLMs.