Improve Variational Autoencoder for Text Generationwith Discrete Latent Bottleneck

TL;DR

This paper introduces a discretized bottleneck in variational autoencoders to improve text generation by encouraging meaningful latent representations, leading to better interpretability and performance across various NLP tasks.

Contribution

The paper proposes a novel discretized latent bottleneck in VAEs that enhances semantic modeling and interpretability in text generation tasks.

Findings

Improved text generation quality across multiple NLP tasks.

Enhanced interpretability of latent structures.

Demonstrated efficiency and effectiveness empirically.

Abstract

Variational autoencoders (VAEs) are essential tools in end-to-end representation learning. However, the sequential text generation common pitfall with VAEs is that the model tends to ignore latent variables with a strong auto-regressive decoder. In this paper, we propose a principled approach to alleviate this issue by applying a discretized bottleneck to enforce an implicit latent feature matching in a more compact latent space. We impose a shared discrete latent space where each input is learned to choose a combination of latent atoms as a regularized latent representation. Our model endows a promising capability to model underlying semantics of discrete sequences and thus provide more interpretative latent structures. Empirically, we demonstrate our model's efficiency and effectiveness on a broad range of tasks, including language modeling, unaligned text style transfer, dialog response generation, and neural machine translation.