CoNT: Contrastive Neural Text Generation

TL;DR

CoNT introduces a contrastive learning framework for neural text generation, significantly improving performance across multiple tasks by addressing key bottlenecks in contrastive example construction, loss choice, and decoding strategies.

Contribution

The paper proposes CoNT, a novel contrastive neural text generation framework that enhances performance by optimizing contrastive example construction, loss functions, and decoding strategies.

Findings

Outperforms traditional training on 10 benchmarks

Achieves state-of-the-art results in summarization and code comment generation

Surpasses previous contrastive learning methods by significant margins

Abstract

Recently, contrastive learning attracts increasing interests in neural text generation as a new solution to alleviate the exposure bias problem. It introduces a sequence-level training signal which is crucial to generation tasks that always rely on auto-regressive decoding. However, previous methods using contrastive learning in neural text generation usually lead to inferior performance. In this paper, we analyse the underlying reasons and propose a new Contrastive Neural Text generation framework, CoNT. CoNT addresses bottlenecks that prevent contrastive learning from being widely adopted in generation tasks from three aspects -- the construction of contrastive examples, the choice of the contrastive loss, and the strategy in decoding. We validate CoNT on five generation tasks with ten benchmarks, including machine translation, summarization, code comment generation, data-to-text generation and commonsense generation. Experimental results show that CoNT clearly outperforms the conventional training framework on all the ten benchmarks with a convincing margin. Especially, CoNT surpasses previous the most competitive contrastive learning method for text generation, by 1.50 BLEU on machine translation and 1.77 ROUGE-1 on summarization, respectively. It achieves new state-of-the-art on summarization, code comment generation (without external data) and data-to-text generation.