Learning from LDA using Deep Neural Networks

TL;DR

This paper introduces a method where a deep neural network is trained to mimic LDA's topic inference, significantly speeding up the process while maintaining similar performance.

Contribution

It presents a novel approach using LDA to supervise DNN training, enabling fast approximation of LDA inference through transfer learning.

Findings

DNN can effectively learn LDA's behavior

Inference speed is increased by tens or hundreds of times

Maintains comparable accuracy to LDA in topic inference

Abstract

Latent Dirichlet Allocation (LDA) is a three-level hierarchical Bayesian model for topic inference. In spite of its great success, inferring the latent topic distribution with LDA is time-consuming. Motivated by the transfer learning approach proposed by~\newcite{hinton2015distilling}, we present a novel method that uses LDA to supervise the training of a deep neural network (DNN), so that the DNN can approximate the costly LDA inference with less computation. Our experiments on a document classification task show that a simple DNN can learn the LDA behavior pretty well, while the inference is speeded up tens or hundreds of times.