Distribution Prompting: Understanding the Expressivity of Language Models Through the Next-Token Distributions They Can Produce

TL;DR

This paper investigates the range of probability distributions that autoregressive language models can produce, revealing that some distributions are easier to approximate and providing insights into their expressiveness and limitations.

Contribution

It systematically analyzes the expressivity of language models by examining how well they can approximate various target distributions using prompt tuning.

Findings

Distributions with very low or high entropy are easier to approximate.

Distributions with outlier tokens are easier to approximate within the same entropy level.

Target distributions generated by LMs are easier to approximate than random distributions.

Abstract

Autoregressive neural language models (LMs) generate a probability distribution over tokens at each time step given a prompt. In this work, we attempt to systematically understand the probability distributions that LMs can produce, showing that some distributions are significantly harder to elicit than others. Specifically, for any target next-token distribution over the vocabulary, we attempt to find a prompt that induces the LM to output a distribution as close as possible to the target, using either soft or hard gradient-based prompt tuning. We find that (1) in general, distributions with very low or very high entropy are easier to approximate than those with moderate entropy; (2) among distributions with the same entropy, those containing ''outlier tokens'' are easier to approximate; (3) target distributions generated by LMs -- even LMs with different tokenizers -- are easier to approximate than randomly chosen targets. These results offer insights into the expressiveness of LMs and the challenges of using them as probability distribution proposers.