WHAI: Weibull Hybrid Autoencoding Inference for Deep Topic Modeling

TL;DR

This paper introduces WHAI, a scalable deep topic modeling method that combines stochastic-gradient MCMC and autoencoding variational Bayes, using Weibull distributions for efficient inference in large text corpora.

Contribution

It develops a novel hybrid inference network for deep latent Dirichlet allocation utilizing Weibull distributions for improved efficiency and scalability.

Findings

Demonstrates effectiveness on large corpora

Achieves faster inference with comparable accuracy

Outperforms existing deep topic models in scalability

Abstract

To train an inference network jointly with a deep generative topic model, making it both scalable to big corpora and fast in out-of-sample prediction, we develop Weibull hybrid autoencoding inference (WHAI) for deep latent Dirichlet allocation, which infers posterior samples via a hybrid of stochastic-gradient MCMC and autoencoding variational Bayes. The generative network of WHAI has a hierarchy of gamma distributions, while the inference network of WHAI is a Weibull upward-downward variational autoencoder, which integrates a deterministic-upward deep neural network, and a stochastic-downward deep generative model based on a hierarchy of Weibull distributions. The Weibull distribution can be used to well approximate a gamma distribution with an analytic Kullback-Leibler divergence, and has a simple reparameterization via the uniform noise, which help efficiently compute the gradients of the evidence lower bound with respect to the parameters of the inference network. The effectiveness and efficiency of WHAI are illustrated with experiments on big corpora.