Neural Text Generation: Past, Present and Beyond

TL;DR

This paper systematically reviews neural text generation models, highlighting their evolution, comparing different techniques, and benchmarking their performance on standard datasets to understand their strengths and limitations.

Contribution

It provides a comprehensive survey of recent neural text generation methods, including reinforcement learning, GANs, and re-parametrization tricks, and evaluates their empirical performance.

Findings

Reinforcement learning improves generation diversity.

GAN-based models face challenges with training stability.

Benchmarking reveals trade-offs between quality and diversity.

Abstract

This paper presents a systematic survey on recent development of neural text generation models. Specifically, we start from recurrent neural network language models with the traditional maximum likelihood estimation training scheme and point out its shortcoming for text generation. We thus introduce the recently proposed methods for text generation based on reinforcement learning, re-parametrization tricks and generative adversarial nets (GAN) techniques. We compare different properties of these models and the corresponding techniques to handle their common problems such as gradient vanishing and generation diversity. Finally, we conduct a benchmarking experiment with different types of neural text generation models on two well-known datasets and discuss the empirical results along with the aforementioned model properties.