An Invariant Latent Space Perspective on Language Model Inversion

TL;DR

This paper introduces Inv^2A, a novel method for language model inversion that leverages invariant latent space properties to improve prompt recovery, highlighting privacy risks and limitations of existing defenses.

Contribution

It proposes the Invariant Latent Space Hypothesis and a lightweight inverse encoder approach, achieving better performance with less data compared to existing methods.

Findings

Inv^2A outperforms baselines by 4.77% BLEU score on average.

The method reduces dependence on large inverse corpora.

Existing defenses offer limited protection against inversion.

Abstract

Language model inversion (LMI), i.e., recovering hidden prompts from outputs, emerges as a concrete threat to user privacy and system security. We recast LMI as reusing the LLM's own latent space and propose the Invariant Latent Space Hypothesis (ILSH): (1) diverse outputs from the same source prompt should preserve consistent semantics (source invariance), and (2) input<->output cyclic mappings should be self-consistent within a shared latent space (cyclic invariance). Accordingly, we present Inv^2A, which treats the LLM as an invariant decoder and learns only a lightweight inverse encoder that maps outputs to a denoised pseudo-representation. When multiple outputs are available, they are sparsely concatenated at the representation layer to increase information density. Training proceeds in two stages: contrastive alignment (source invariance) and supervised reinforcement (cyclic invariance). An optional training-free neighborhood search can refine local performance. Across 9 datasets covering user and system prompt scenarios, Inv^2A outperforms baselines by an average of 4.77% BLEU score while reducing dependence on large inverse corpora. Our analysis further shows that prevalent defenses provide limited protection, underscoring the need for stronger strategies. The source code and data involved in this paper can be found in https://github.com/yyy01/Invariant_Attacker.