Jailbreaking LLMs' Safeguard with Universal Magic Words for Text Embedding Models

TL;DR

This paper identifies a bias in text embedding models that can be exploited using universal magic words to bypass safeguards in large language models, leading to potential security breaches, and proposes methods to defend against such attacks.

Contribution

It introduces a novel attack method using universal magic words to manipulate text embeddings and bypass safeguards, along with effective defense strategies to mitigate this security risk.

Findings

Magic words significantly degrade safeguard performance.

Attacks cause harmful outputs in real-world chatbots.

Defense methods effectively reduce embedding bias.

Abstract

The security issue of large language models (LLMs) has gained wide attention recently, with various defense mechanisms developed to prevent harmful output, among which safeguards based on text embedding models serve as a fundamental defense. Through testing, we discover that the output distribution of text embedding models is severely biased with a large mean. Inspired by this observation, we propose novel, efficient methods to search for **universal magic words** that attack text embedding models. Universal magic words as suffixes can shift the embedding of any text towards the bias direction, thus manipulating the similarity of any text pair and misleading safeguards. Attackers can jailbreak the safeguards by appending magic words to user prompts and requiring LLMs to end answers with magic words. Experiments show that magic word attacks significantly degrade safeguard performance on JailbreakBench, cause real-world chatbots to produce harmful outputs in full-pipeline attacks, and generalize across input/output texts, models, and languages. To eradicate this security risk, we also propose defense methods against such attacks, which can correct the bias of text embeddings and improve downstream performance in a train-free manner.