PSM: Prompt Sensitivity Minimization via LLM-Guided Black-Box Optimization

TL;DR

This paper proposes a lightweight, black-box optimization framework to harden system prompts in LLMs by adding protective layers, significantly reducing prompt leakage from adversarial attacks while maintaining task utility.

Contribution

It introduces a formal utility-constrained optimization approach using an LLM as an optimizer to generate shield prompts that minimize leakage without sacrificing performance.

Findings

Optimized SHIELDs significantly reduce prompt leakage against extraction attacks.

The method outperforms existing defenses in effectiveness.

It maintains high task utility with minimal overhead.

Abstract

System prompts are critical for guiding the behavior of Large Language Models (LLMs), yet they often contain proprietary logic or sensitive information, making them a prime target for extraction attacks. Adversarial queries can successfully elicit these hidden instructions, posing significant security and privacy risks. Existing defense mechanisms frequently rely on heuristics, incur substantial computational overhead, or are inapplicable to models accessed via black-box APIs. This paper introduces a novel framework for hardening system prompts through shield appending, a lightweight approach that adds a protective textual layer to the original prompt. Our core contribution is the formalization of prompt hardening as a utility-constrained optimization problem. We leverage an LLM-as-optimizer to search the space of possible SHIELDs, seeking to minimize a leakage metric derived from a suite of adversarial attacks, while simultaneously preserving task utility above a specified threshold, measured by semantic fidelity to baseline outputs. This black-box, optimization-driven methodology is lightweight and practical, requiring only API access to the target and optimizer LLMs. We demonstrate empirically that our optimized SHIELDs significantly reduce prompt leakage against a comprehensive set of extraction attacks, outperforming established baseline defenses without compromising the model's intended functionality. Our work presents a paradigm for developing robust, utility-aware defenses in the escalating landscape of LLM security. The code is made public on the following link: https://github.com/psm-defense/psm