Latent Space Energy-Based Model of Symbol-Vector Coupling for Text Generation and Classification

TL;DR

This paper introduces a latent space energy-based prior model for text generation and classification that couples continuous and symbolic representations, enabling high-quality text synthesis and effective categorization, even with limited supervision.

Contribution

The paper presents a novel latent space energy-based model that jointly learns text generation and classification, incorporating a coupling mechanism for continuous and symbolic vectors, and supports unsupervised and semi-supervised learning.

Findings

Learned well-structured, meaningful latent space

Generated high-quality, diverse, and interpretable text

Achieved effective text classification

Abstract

We propose a latent space energy-based prior model for text generation and classification. The model stands on a generator network that generates the text sequence based on a continuous latent vector. The energy term of the prior model couples a continuous latent vector and a symbolic one-hot vector, so that discrete category can be inferred from the observed example based on the continuous latent vector. Such a latent space coupling naturally enables incorporation of information bottleneck regularization to encourage the continuous latent vector to extract information from the observed example that is informative of the underlying category. In our learning method, the symbol-vector coupling, the generator network and the inference network are learned jointly. Our model can be learned in an unsupervised setting where no category labels are provided. It can also be learned in semi-supervised setting where category labels are provided for a subset of training examples. Our experiments demonstrate that the proposed model learns well-structured and meaningful latent space, which (1) guides the generator to generate text with high quality, diversity, and interpretability, and (2) effectively classifies text.