Multi-Amateur Contrastive Decoding for Text Generation

TL;DR

Multi-Amateur Contrastive Decoding (MACD) enhances open-ended text generation by using multiple amateur models to better identify and avoid undesirable outputs, improving fluency, coherence, and diversity without extra training.

Contribution

The paper introduces MACD, a novel extension of Contrastive Decoding that employs multiple amateur models for more comprehensive and controllable text generation.

Findings

MACD outperforms traditional decoding methods in multiple domains.

MACD improves fluency, coherence, and diversity of generated text.

MACD enables controllable generation with targeted stylistic biases.

Abstract

Contrastive Decoding (CD) has emerged as an effective inference-time strategy for enhancing open-ended text generation by exploiting the divergence in output probabilities between a large expert language model and a smaller amateur model. Although CD improves coherence and fluency, its dependence on a single amateur restricts its capacity to capture the diverse and multifaceted failure modes of language generation, such as repetition, hallucination, and stylistic drift. This paper proposes Multi-Amateur Contrastive Decoding (MACD), a generalization of the CD framework that employs an ensemble of amateur models to more comprehensively characterize undesirable generation patterns. MACD integrates contrastive signals through both averaging and consensus penalization mechanisms and extends the plausibility constraint to operate effectively in the multi-amateur setting. Furthermore, the framework enables controllable generation by incorporating amateurs with targeted stylistic or content biases. Experimental results across multiple domains, such as news, encyclopedic, and narrative, demonstrate that MACD consistently surpasses conventional decoding methods and the original CD approach in terms of fluency, coherence, diversity, and adaptability, all without requiring additional training or fine-tuning.