Adaptive Contrastive Search: Uncertainty-Guided Decoding for Open-Ended Text Generation

TL;DR

This paper introduces adaptive contrastive search, a decoding method that uses uncertainty-guided penalties to improve creativity, diversity, and coherence in open-ended text generation from large language models.

Contribution

It proposes a novel adaptive decoding strategy that extends contrastive search by incorporating uncertainty-based penalties, enhancing text quality and diversity.

Findings

Improves coherence and diversity of generated text

Enhances performance across different models and datasets

Publicly available code and datasets

Abstract

Decoding from the output distributions of large language models to produce high-quality text is a complex challenge in language modeling. Various approaches, such as beam search, sampling with temperature, $k-$sampling, nucleus $p-$sampling, typical decoding, contrastive decoding, and contrastive search, have been proposed to address this problem, aiming to improve coherence, diversity, as well as resemblance to human-generated text. In this study, we introduce adaptive contrastive search, a novel decoding strategy extending contrastive search by incorporating an adaptive degeneration penalty, guided by the estimated uncertainty of the model at each generation step. This strategy is designed to enhance both the creativity and diversity of the language modeling process while at the same time producing coherent and high-quality generated text output. Our findings indicate performance enhancement in both aspects, across different model architectures and datasets, underscoring the effectiveness of our method in text generation tasks. Our code base, datasets, and models are publicly available.