Eval all, trust a few, do wrong to none: Comparing sentence generation models

TL;DR

This paper conducts a comprehensive evaluation of neural generative models for text, focusing on both sample quality and reconstruction accuracy, proposing a rigorous evaluation protocol to standardize comparisons.

Contribution

It introduces a thorough evaluation framework for neural text generation models, emphasizing the importance of reconstruction error alongside sampling quality.

Findings

Evaluation protocol covers automatic and human metrics

Reconstruction error is crucial for assessing model performance

Proposes a new standard for comparing generative models

Abstract

In this paper, we study recent neural generative models for text generation related to variational autoencoders. Previous works have employed various techniques to control the prior distribution of the latent codes in these models, which is important for sampling performance, but little attention has been paid to reconstruction error. In our study, we follow a rigorous evaluation protocol using a large set of previously used and novel automatic and human evaluation metrics, applied to both generated samples and reconstructions. We hope that it will become the new evaluation standard when comparing neural generative models for text.