Variational Autoencoder with Disentanglement Priors for Low-Resource Task-Specific Natural Language Generation

TL;DR

This paper introduces VAE-DPRIOR, a variational autoencoder with novel disentanglement priors, to improve low-resource task-specific natural language generation by enabling semantic diversity and better generalization across tasks.

Contribution

The paper proposes a new disentangled prior for VAEs that enhances low-resource NLP tasks by enabling semantic diversity and compositional generalization without complex regularizations.

Findings

Outperforms baselines in zero/few-shot data augmentation

Achieves superior results in style transfer tasks

Disentangles representations effectively without additional regularizations

Abstract

In this paper, we propose a variational autoencoder with disentanglement priors, VAE-DPRIOR, for task-specific natural language generation with none or a handful of task-specific labeled examples. In order to tackle compositional generalization across tasks, our model performs disentangled representation learning by introducing a conditional prior for the latent content space and another conditional prior for the latent label space. Both types of priors satisfy a novel property called $\epsilon$-disentangled. We show both empirically and theoretically that the novel priors can disentangle representations even without specific regularizations as in the prior work. The content prior enables directly sampling diverse content representations from the content space learned from the seen tasks, and fuse them with the representations of novel tasks for generating semantically diverse texts in the low-resource settings. Our extensive experiments demonstrate the superior performance of our model over competitive baselines in terms of i) data augmentation in continuous zero/few-shot learning, and ii) text style transfer in the few-shot setting.