Variational Inference In Pachinko Allocation Machines

TL;DR

This paper introduces an efficient amortized variational inference method for Pachinko Allocation Machines, enabling faster and more coherent topic modeling while facilitating exploration of diverse PAM architectures.

Contribution

It presents a novel deep inference network for PAM that improves inference speed and topic coherence, expanding the scope of feasible PAM architectures.

Findings

More coherent topics than state-of-the-art methods

Order of magnitude faster inference

Enables exploration of diverse PAM architectures

Abstract

The Pachinko Allocation Machine (PAM) is a deep topic model that allows representing rich correlation structures among topics by a directed acyclic graph over topics. Because of the flexibility of the model, however, approximate inference is very difficult. Perhaps for this reason, only a small number of potential PAM architectures have been explored in the literature. In this paper we present an efficient and flexible amortized variational inference method for PAM, using a deep inference network to parameterize the approximate posterior distribution in a manner similar to the variational autoencoder. Our inference method produces more coherent topics than state-of-art inference methods for PAM while being an order of magnitude faster, which allows exploration of a wider range of PAM architectures than have previously been studied.