On-the-Fly Attention Modulation for Neural Generation

TL;DR

This paper introduces on-the-fly attention modulation, a method to improve neural text generation by injecting priors into attention during inference, resulting in more fluent, creative, and commonsense outputs.

Contribution

It proposes a novel inference-time attention modulation technique that enhances language model outputs without retraining, addressing degeneration issues.

Findings

Reduces sentence-level repetition in generated text

Improves fluency, creativity, and commonsense reasoning

Effective across multiple benchmarks

Abstract

Despite considerable advancements with deep neural language models (LMs), neural text generation still suffers from degeneration: the generated text is repetitive, generic, self-contradictory, and often lacks commonsense. Our analyses on sentence-level attention patterns in LMs reveal that neural degeneration may be associated with insufficient learning of task-specific characteristics by the attention mechanism. This finding motivates on-the-fly attention modulation -- a simple but effective method that enables the injection of priors into attention computation during inference. Automatic and human evaluation results on three text generation benchmarks demonstrate that attention modulation helps LMs generate text with enhanced fluency, creativity, and commonsense reasoning, in addition to significantly reduce sentence-level repetition.