Human-Interpretable Adversarial Prompt Attack on Large Language Models with Situational Context

TL;DR

This paper introduces a human-understandable adversarial prompt attack method on large language models that uses situational context to craft effective prompts without gradient access, revealing significant vulnerabilities.

Contribution

The study presents a novel approach to convert nonsensical prompt injections into meaningful, situationally relevant attacks using only LLMs, enhancing understanding of LLM vulnerabilities.

Findings

Successful attacks on multiple LLMs with as few as one attempt

Attacks transfer across different LLM architectures

Effective attack without gradient access using situational rewriting

Abstract

Previous research on testing the vulnerabilities in Large Language Models (LLMs) using adversarial attacks has primarily focused on nonsensical prompt injections, which are easily detected upon manual or automated review (e.g., via byte entropy). However, the exploration of innocuous human-understandable malicious prompts augmented with adversarial injections remains limited. In this research, we explore converting a nonsensical suffix attack into a sensible prompt via a situation-driven contextual re-writing. This allows us to show suffix conversion without any gradients, using only LLMs to perform the attacks, and thus better understand the scope of possible risks. We combine an independent, meaningful adversarial insertion and situations derived from movies to check if this can trick an LLM. The situations are extracted from the IMDB dataset, and prompts are defined following a few-shot chain-of-thought prompting. Our approach demonstrates that a successful situation-driven attack can be executed on both open-source and proprietary LLMs. We find that across many LLMs, as few as 1 attempt produces an attack and that these attacks transfer between LLMs.