De-Anonymizing Text by Fingerprinting Language Generation

TL;DR

This paper reveals that nucleus sampling in language generation can unintentionally leak user-typed texts through unique fingerprints, enabling de-anonymization and raising security concerns in ML systems.

Contribution

It introduces a novel fingerprinting method based on nucleus sampling sizes and demonstrates its potential for de-anonymizing texts, highlighting security vulnerabilities in ML-generated content.

Findings

Nucleus sampling sizes form unique fingerprints for English text sequences.

Attackers can infer user-typed text via side-channel measurements like cache access times.

The study discusses potential defenses against fingerprinting attacks.

Abstract

Components of machine learning systems are not (yet) perceived as security hotspots. Secure coding practices, such as ensuring that no execution paths depend on confidential inputs, have not yet been adopted by ML developers. We initiate the study of code security of ML systems by investigating how nucleus sampling---a popular approach for generating text, used for applications such as auto-completion---unwittingly leaks texts typed by users. Our main result is that the series of nucleus sizes for many natural English word sequences is a unique fingerprint. We then show how an attacker can infer typed text by measuring these fingerprints via a suitable side channel (e.g., cache access times), explain how this attack could help de-anonymize anonymous texts, and discuss defenses.