Raccoon: Prompt Extraction Benchmark of LLM-Integrated Applications

TL;DR

The paper introduces Raccoon, a comprehensive benchmark for evaluating the vulnerability of LLMs to prompt extraction attacks, including diverse attack types and defenses, to improve robustness assessment.

Contribution

It presents the first extensive benchmark evaluating LLM susceptibility to prompt theft, with novel dual-scenario assessment and a wide range of attack and defense strategies.

Findings

Models are generally vulnerable without defenses.

OpenAI models show resilience with proper defenses.

The benchmark covers 14 attack categories and defense mechanisms.

Abstract

With the proliferation of LLM-integrated applications such as GPT-s, millions are deployed, offering valuable services through proprietary instruction prompts. These systems, however, are prone to prompt extraction attacks through meticulously designed queries. To help mitigate this problem, we introduce the Raccoon benchmark which comprehensively evaluates a model's susceptibility to prompt extraction attacks. Our novel evaluation method assesses models under both defenseless and defended scenarios, employing a dual approach to evaluate the effectiveness of existing defenses and the resilience of the models. The benchmark encompasses 14 categories of prompt extraction attacks, with additional compounded attacks that closely mimic the strategies of potential attackers, alongside a diverse collection of defense templates. This array is, to our knowledge, the most extensive compilation of prompt theft attacks and defense mechanisms to date. Our findings highlight universal susceptibility to prompt theft in the absence of defenses, with OpenAI models demonstrating notable resilience when protected. This paper aims to establish a more systematic benchmark for assessing LLM robustness against prompt extraction attacks, offering insights into their causes and potential countermeasures. Resources of Raccoon are publicly available at https://github.com/M0gician/RaccoonBench.