Extracting Prompts by Inverting LLM Outputs

TL;DR

This paper introduces output2prompt, a black-box method for extracting prompts from language model outputs without needing access to model internals, improving efficiency and transferability.

Contribution

The paper presents a novel zero-shot prompt extraction technique that operates solely on model outputs and employs sparse encoding for memory efficiency.

Findings

Effective prompt extraction across multiple LLMs

Zero-shot transferability demonstrated

Memory-efficient sparse encoding technique

Abstract

We consider the problem of language model inversion: given outputs of a language model, we seek to extract the prompt that generated these outputs. We develop a new black-box method, output2prompt, that learns to extract prompts without access to the model's logits and without adversarial or jailbreaking queries. In contrast to previous work, output2prompt only needs outputs of normal user queries. To improve memory efficiency, output2prompt employs a new sparse encoding techique. We measure the efficacy of output2prompt on a variety of user and system prompts and demonstrate zero-shot transferability across different LLMs.