Lightweight Decoding Strategies for Increasing Specificity

TL;DR

This paper introduces two unsupervised decoding strategies that enhance the specificity of language model outputs, making responses more detailed and less generic, with minimal loss of sensibility.

Contribution

It proposes novel decoding methods based on word-frequency and mutual information to improve output specificity in language models.

Findings

Both strategies increase output specificity in prompt completion tasks.

Strategies cause only modest decreases in sensibility.

Applicable to summarization for more specific summaries.

Abstract

Language models are known to produce vague and generic outputs. We propose two unsupervised decoding strategies based on either word-frequency or point-wise mutual information to increase the specificity of any model that outputs a probability distribution over its vocabulary at generation time. We test the strategies in a prompt completion task; with human evaluations, we find that both strategies increase the specificity of outputs with only modest decreases in sensibility. We also briefly present a summarization use case, where these strategies can produce more specific summaries.