Ignore Previous Prompt: Attack Techniques For Language Models

TL;DR

This paper introduces PromptInject, a framework demonstrating how malicious prompts can exploit GPT-3's vulnerabilities, revealing significant risks in deploying large language models in real-world applications.

Contribution

The paper presents PromptInject, a novel method for generating adversarial prompts that expose vulnerabilities in GPT-3, highlighting the ease of malicious exploitation.

Findings

GPT-3 can be misaligned by simple handcrafted prompts

Attack techniques like goal hijacking and prompt leaking are effective

Low-aptitude agents can exploit GPT-3's stochastic nature

Abstract

Transformer-based large language models (LLMs) provide a powerful foundation for natural language tasks in large-scale customer-facing applications. However, studies that explore their vulnerabilities emerging from malicious user interaction are scarce. By proposing PromptInject, a prosaic alignment framework for mask-based iterative adversarial prompt composition, we examine how GPT-3, the most widely deployed language model in production, can be easily misaligned by simple handcrafted inputs. In particular, we investigate two types of attacks -- goal hijacking and prompt leaking -- and demonstrate that even low-aptitude, but sufficiently ill-intentioned agents, can easily exploit GPT-3's stochastic nature, creating long-tail risks. The code for PromptInject is available at https://github.com/agencyenterprise/PromptInject.