Logits of API-Protected LLMs Leak Proprietary Information

TL;DR

This paper demonstrates that proprietary API-protected LLMs leak significant non-public information through their logits due to the softmax bottleneck, enabling various extraction and auditing capabilities with minimal cost.

Contribution

It reveals a novel vulnerability in API-protected LLMs caused by the softmax bottleneck, enabling information leakage and model analysis from limited API queries.

Findings

Estimated GPT-3.5-turbo embedding size is about 4096.

Able to recover full vocabulary outputs at low cost.

Can identify model updates and source LLMs effectively.

Abstract

Large language model (LLM) providers often hide the architectural details and parameters of their proprietary models by restricting public access to a limited API. In this work we show that, with only a conservative assumption about the model architecture, it is possible to learn a surprisingly large amount of non-public information about an API-protected LLM from a relatively small number of API queries (e.g., costing under $1000 USD for OpenAI's gpt-3.5-turbo). Our findings are centered on one key observation: most modern LLMs suffer from a softmax bottleneck, which restricts the model outputs to a linear subspace of the full output space. We exploit this fact to unlock several capabilities, including (but not limited to) obtaining cheap full-vocabulary outputs, auditing for specific types of model updates, identifying the source LLM given a single full LLM output, and even efficiently discovering the LLM's hidden size. Our empirical investigations show the effectiveness of our methods, which allow us to estimate the embedding size of OpenAI's gpt-3.5-turbo to be about 4096. Lastly, we discuss ways that LLM providers can guard against these attacks, as well as how these capabilities can be viewed as a feature (rather than a bug) by allowing for greater transparency and accountability.