System Prompt Extraction Attacks and Defenses in Large Language Models

TL;DR

This paper systematically studies the vulnerability of Large Language Models' system prompts to extraction attacks, proposing novel attack methods and defense techniques, and evaluating their effectiveness through comprehensive experiments.

Contribution

It introduces the SPE-LLM framework, including novel adversarial queries and defense strategies, to address the security risks of prompt extraction in LLMs.

Findings

Effective adversarial queries for prompt extraction

Proposed defense techniques reduce attack success

Validated framework across multiple benchmarks

Abstract

The system prompt in Large Language Models (LLMs) plays a pivotal role in guiding model behavior and response generation. Often containing private configuration details, user roles, and operational instructions, the system prompt has become an emerging attack target. Recent studies have shown that LLM system prompts are highly susceptible to extraction attacks through meticulously designed queries, raising significant privacy and security concerns. Despite the growing threat, there is a lack of systematic studies of system prompt extraction attacks and defenses. In this paper, we present a comprehensive framework, SPE-LLM, to systematically evaluate System Prompt Extraction attacks and defenses in LLMs. First, we design a set of novel adversarial queries that effectively extract system prompts in state-of-the-art (SOTA) LLMs, demonstrating the severe risks of LLM system prompt extraction attacks. Second, we propose three defense techniques to mitigate system prompt extraction attacks in LLMs, providing practical solutions for secure LLM deployments. Third, we introduce a set of rigorous evaluation metrics to accurately quantify the severity of system prompt extraction attacks in LLMs and conduct comprehensive experiments across multiple benchmark datasets, which validates the efficacy of our proposed SPE-LLM framework.