Disentangling Generative Factors in Natural Language with Discrete Variational Autoencoders

TL;DR

This paper introduces a discrete variational autoencoder approach for learning disentangled, interpretable representations in NLP by modeling linguistic features as discrete variables, outperforming existing methods.

Contribution

It proposes a novel discrete VAE model tailored for textual data, addressing limitations of continuous models in capturing discrete linguistic features.

Findings

Outperforms continuous and discrete baselines on disentanglement benchmarks

Achieves better interpretability of linguistic features

Improves text style transfer performance

Abstract

The ability of learning disentangled representations represents a major step for interpretable NLP systems as it allows latent linguistic features to be controlled. Most approaches to disentanglement rely on continuous variables, both for images and text. We argue that despite being suitable for image datasets, continuous variables may not be ideal to model features of textual data, due to the fact that most generative factors in text are discrete. We propose a Variational Autoencoder based method which models language features as discrete variables and encourages independence between variables for learning disentangled representations. The proposed model outperforms continuous and discrete baselines on several qualitative and quantitative benchmarks for disentanglement as well as on a text style transfer downstream application.