BeamClean: Language Aware Embedding Reconstruction

TL;DR

BeamClean is a novel inversion attack that effectively reconstructs original input embeddings from obfuscated embeddings in language models by leveraging language priors, surpassing naive methods and emphasizing the need for stronger defenses.

Contribution

This work introduces BeamClean, a new inversion attack that jointly estimates noise and decodes tokens using language-model priors, improving over existing methods.

Findings

BeamClean outperforms naive distance-based attacks on obfuscated embeddings.

It effectively reconstructs token sequences under Laplacian and Gaussian obfuscation.

The results demonstrate the need for more advanced, input-dependent obfuscation defenses.

Abstract

In this work, we consider an inversion attack on the obfuscated input embeddings sent to a language model on a server, where the adversary has no access to the language model or the obfuscation mechanism and sees only the obfuscated embeddings along with the model's embedding table. We propose BeamClean, an inversion attack that jointly estimates the noise parameters and decodes token sequences by integrating a language-model prior. Against Laplacian and Gaussian obfuscation mechanisms, BeamClean always surpasses naive distance-based attacks. This work highlights the necessity for and robustness of more advanced learned, input-dependent methods.