On-the-Fly Attention Modulation for Neural Generation
Yue Dong, Chandra Bhagavatula, Ximing Lu, Jena D. Hwang, Antoine, Bosselut, Jackie Chi Kit Cheung, Yejin Choi

TL;DR
This paper introduces on-the-fly attention modulation, a method to improve neural text generation by injecting priors into attention during inference, resulting in more fluent, creative, and commonsense outputs.
Contribution
It proposes a novel inference-time attention modulation technique that enhances language model outputs without retraining, addressing degeneration issues.
Findings
Reduces sentence-level repetition in generated text
Improves fluency, creativity, and commonsense reasoning
Effective across multiple benchmarks
Abstract
Despite considerable advancements with deep neural language models (LMs), neural text generation still suffers from degeneration: the generated text is repetitive, generic, self-contradictory, and often lacks commonsense. Our analyses on sentence-level attention patterns in LMs reveal that neural degeneration may be associated with insufficient learning of task-specific characteristics by the attention mechanism. This finding motivates on-the-fly attention modulation -- a simple but effective method that enables the injection of priors into attention computation during inference. Automatic and human evaluation results on three text generation benchmarks demonstrate that attention modulation helps LMs generate text with enhanced fluency, creativity, and commonsense reasoning, in addition to significantly reduce sentence-level repetition.
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsTopic Modeling · Natural Language Processing Techniques · Text Readability and Simplification
