Defending Large Language Models against Jailbreak Attacks via Semantic Smoothing

TL;DR

This paper introduces SEMANTICSMOOTH, a novel defense method that enhances large language models' robustness against semantic jailbreak attacks by aggregating predictions over semantically transformed inputs, achieving state-of-the-art results.

Contribution

We propose SEMANTICSMOOTH, a smoothing-based defense that improves robustness against semantic jailbreak attacks without sacrificing nominal performance.

Findings

Achieves state-of-the-art robustness against GCG, PAIR, and AutoDAN attacks.

Maintains strong performance on instruction following benchmarks.

Codes will be publicly available.

Abstract

Aligned large language models (LLMs) are vulnerable to jailbreaking attacks, which bypass the safeguards of targeted LLMs and fool them into generating objectionable content. While initial defenses show promise against token-based threat models, there do not exist defenses that provide robustness against semantic attacks and avoid unfavorable trade-offs between robustness and nominal performance. To meet this need, we propose SEMANTICSMOOTH, a smoothing-based defense that aggregates the predictions of multiple semantically transformed copies of a given input prompt. Experimental results demonstrate that SEMANTICSMOOTH achieves state-of-the-art robustness against GCG, PAIR, and AutoDAN attacks while maintaining strong nominal performance on instruction following benchmarks such as InstructionFollowing and AlpacaEval. The codes will be publicly available at https://github.com/UCSB-NLP-Chang/SemanticSmooth.