Reverse Prompt Engineering

TL;DR

This paper introduces a training-free, black-box method for reverse prompt engineering that reconstructs prompts from limited model outputs using a genetic algorithm-inspired optimization, outperforming existing methods in coherence and semantic alignment.

Contribution

The paper presents a novel, resource-efficient framework for prompt reconstruction that does not require extensive data or training, unlike prior approaches.

Findings

Achieves high-quality prompt recovery from limited outputs.

Produces prompts more semantically aligned with originals.

Demonstrates strong potential for generating high-quality data from perturbed prompts.

Abstract

We explore a new language model inversion problem under strict black-box, zero-shot, and limited data conditions. We propose a novel training-free framework that reconstructs prompts using only a limited number of text outputs from a language model. Existing methods rely on the availability of a large number of outputs for both training and inference, an assumption that is unrealistic in the real world, and they can sometimes produce garbled text. In contrast, our approach, which relies on limited resources, consistently yields coherent and semantically meaningful prompts. Our framework leverages a large language model together with an optimization process inspired by the genetic algorithm to effectively recover prompts. Experimental results on several datasets derived from public sources indicate that our approach achieves high-quality prompt recovery and generates prompts more semantically and functionally aligned with the originals than current state-of-the-art methods. Additionally, use-case studies introduced demonstrate the method's strong potential for generating high-quality text data on perturbed prompts.