Truncation Sampling as Language Model Desmoothing

TL;DR

This paper introduces a new truncation sampling method called η-sampling that improves the quality and plausibility of long text samples from neural language models by better estimating the true distribution.

Contribution

The work provides a new theoretical framing of truncation as desmoothing and proposes η-sampling, an improved algorithm that adaptively truncates words based on entropy, outperforming previous methods.

Findings

η-sampling produces more plausible long English texts

It better avoids repetitive outputs

It performs well across various test distributions

Abstract

Long samples of text from neural language models can be of poor quality. Truncation sampling algorithms--like top-$p$ or top-$k$ -- address this by setting some words' probabilities to zero at each step. This work provides framing for the aim of truncation, and an improved algorithm for that aim. We propose thinking of a neural language model as a mixture of a true distribution and a smoothing distribution that avoids infinite perplexity. In this light, truncation algorithms aim to perform desmoothing, estimating a subset of the support of the true distribution. Finding a good subset is crucial: we show that top-$p$ unnecessarily truncates high-probability words, for example causing it to truncate all words but Trump for a document that starts with Donald. We introduce $\eta$-sampling, which truncates words below an entropy-dependent probability threshold. Compared to previous algorithms, $\eta$-sampling generates more plausible long English documents according to humans, is better at breaking out of repetition, and behaves more reasonably on a battery of test distributions.