GPT, But Backwards: Exactly Inverting Language Model Outputs

TL;DR

This paper introduces SODA, a search-based algorithm that can precisely invert language model outputs to reconstruct original inputs, revealing security vulnerabilities and scaling across model sizes.

Contribution

The paper presents SODA, the first exact inversion method for language models, capable of reconstructing inputs with high accuracy on both natural and random data.

Findings

Achieves 98% reconstruction on 10-token natural inputs

Achieves 79% reconstruction on 10-token random inputs

Input length and vocabulary size significantly affect reconstruction success

Abstract

The task of reconstructing unknown textual inputs to language models is a fundamental auditing primitive that allows us to assess the model's vulnerability to a range of security issues, including stealing hidden system prompts, detecting backdoors, and leaking private data. Existing inversion works assume access to differing levels of information (e.g. requiring input-output examples, the model parameters, intermediate activations or output logits) but oftentimes fail to fully reconstruct the desired input. In this paper, we present the Sparse One-hot Discrete Adam (SODA) algorithm, a search-based inversion method that can accurately reconstruct the input text, given white-box access to the language model and its output. Our experiments demonstrate for the first time that exact language model inversion is possible on both natural language and random inputs. Indeed, SODA achieves respectively 98% and 79% reconstruction rates on inputs with lengths up to 10 tokens. Furthermore, we show that input length and vocabulary size have a far greater impact on the probability of a successful reconstruction than the size of the language model itself, thus allowing us to scale to models from 33M to 3B parameters.