ProxyPrompt: Securing System Prompts against Prompt Extraction Attacks

TL;DR

This paper introduces ProxyPrompt, a defense mechanism that replaces system prompts with proxies to prevent prompt extraction attacks, significantly improving security without sacrificing task utility.

Contribution

ProxyPrompt is a novel approach that effectively obfuscates system prompts, safeguarding sensitive information against extraction attacks while maintaining task performance.

Findings

ProxyPrompt protects 94.70% of prompts from extraction attacks.

It outperforms existing defenses, which protect only 42.80%.

Evaluations conducted on 264 LLM and prompt pairs.

Abstract

The integration of large language models (LLMs) into a wide range of applications has highlighted the critical role of well-crafted system prompts, which require extensive testing and domain expertise. These prompts enhance task performance but may also encode sensitive information and filtering criteria, posing security risks if exposed. Recent research shows that system prompts are vulnerable to extraction attacks, while existing defenses are either easily bypassed or require constant updates to address new threats. In this work, we introduce ProxyPrompt, a novel defense mechanism that prevents prompt leakage by replacing the original prompt with a proxy. This proxy maintains the original task's utility while obfuscating the extracted prompt, ensuring attackers cannot reproduce the task or access sensitive information. Comprehensive evaluations on 264 LLM and system prompt pairs show that ProxyPrompt protects 94.70% of prompts from extraction attacks, outperforming the next-best defense, which only achieves 42.80%.