Beyond Indistinguishability: Measuring Extraction Risk in LLM APIs

TL;DR

This paper challenges the reliance on indistinguishability as a proxy for privacy in LLM APIs, introducing a new inextractability measure that better captures extraction risks and providing practical evaluation methods.

Contribution

It formalizes a new inextractability definition, develops an extraction risk estimator, and empirically evaluates extraction risks across models and configurations.

Findings

Indistinguishability does not imply inextractability in LLMs.

The proposed estimator effectively measures extraction risk.

Empirical results clarify the connection between extractability and distinguishability.

Abstract

Indistinguishability properties such as differential privacy bounds or low empirically measured membership inference are widely treated as proxies to show a model is sufficiently protected against broader memorization risks. However, we show that indistinguishability properties are neither sufficient nor necessary for preventing data extraction in LLM APIs. We formalize a privacy-game separation between extraction and indistinguishability-based privacy, showing that indistinguishability and inextractability are incomparable: upper-bounding distinguishability does not upper-bound extractability. To address this gap, we introduce $(l, b)$-inextractability as a definition that requires at least $2^b$ expected queries for any black-box adversary to induce the LLM API to emit a protected $l$-gram substring. We instantiate this via a worst-case extraction game and derive a rank-based extraction risk upper bound for targeted exact extraction, as well as extensions to cover untargeted and approximate extraction. The resulting estimator captures the extraction risk over multiple attack trials and prefix adaptations. We show that it can provide a tight and efficient estimation for standard greedy extraction and an upper bound on the probabilistic extraction risk given any decoding configuration. We empirically evaluate extractability across different models, clarifying its connection to distinguishability, demonstrating its advantage over existing extraction risk estimators, and providing actionable mitigation guidelines across model training, API access, and decoding configurations in LLM API deployment. Our code is publicly available at: https://github.com/Emory-AIMS/Inextractability.