DROJ: A Prompt-Driven Attack against Large Language Models

TL;DR

DROJ is a novel prompt optimization method that manipulates embeddings to successfully bypass safety measures in large language models, revealing vulnerabilities and suggesting ways to improve model robustness.

Contribution

The paper introduces DROJ, a new embedding-level prompt optimization technique for adversarial jailbreak attacks on LLMs, demonstrating its effectiveness and proposing mitigation strategies.

Findings

DROJ achieves 100% attack success rate on LLaMA-2-7b-chat.

The attack can bypass safety mechanisms effectively.

Repetitive responses are a side effect of the attack, mitigated by a helpfulness prompt.

Abstract

Large Language Models (LLMs) have demonstrated exceptional capabilities across various natural language processing tasks. Due to their training on internet-sourced datasets, LLMs can sometimes generate objectionable content, necessitating extensive alignment with human feedback to avoid such outputs. Despite massive alignment efforts, LLMs remain susceptible to adversarial jailbreak attacks, which usually are manipulated prompts designed to circumvent safety mechanisms and elicit harmful responses. Here, we introduce a novel approach, Directed Rrepresentation Optimization Jailbreak (DROJ), which optimizes jailbreak prompts at the embedding level to shift the hidden representations of harmful queries towards directions that are more likely to elicit affirmative responses from the model. Our evaluations on LLaMA-2-7b-chat model show that DROJ achieves a 100\% keyword-based Attack Success Rate (ASR), effectively preventing direct refusals. However, the model occasionally produces repetitive and non-informative responses. To mitigate this, we introduce a helpfulness system prompt that enhances the utility of the model's responses. Our code is available at https://github.com/Leon-Leyang/LLM-Safeguard.