Expectation-Propogation for the Generative Aspect Model

TL;DR

This paper introduces an improved inference method for the generative aspect model in text analysis, using Expectation-Propagation to achieve higher accuracy in inference and learning compared to previous variational approaches.

Contribution

It develops an Expectation-Propagation based inference method for the generative aspect model, enhancing accuracy while maintaining computational efficiency.

Findings

Expectation-Propagation improves inference accuracy.

The method outperforms variational approaches on synthetic and real data.

Higher quality parameter estimation achieved with the new approach.

Abstract

The generative aspect model is an extension of the multinomial model for text that allows word probabilities to vary stochastically across documents. Previous results with aspect models have been promising, but hindered by the computational difficulty of carrying out inference and learning. This paper demonstrates that the simple variational methods of Blei et al (2001) can lead to inaccurate inferences and biased learning for the generative aspect model. We develop an alternative approach that leads to higher accuracy at comparable cost. An extension of Expectation-Propagation is used for inference and then embedded in an EM algorithm for learning. Experimental results are presented for both synthetic and real data sets.