Hybrid Variational/Gibbs Collapsed Inference in Topic Models

TL;DR

This paper introduces a hybrid inference algorithm for topic models that combines Gibbs sampling for small counts and variational inference for large counts, improving accuracy without extra computational cost.

Contribution

It presents a novel hybrid inference method that leverages the strengths of both Gibbs sampling and variational inference for Bayesian networks.

Findings

Significantly improves testset perplexity over pure variational inference.

Achieves better accuracy for small counts while maintaining efficiency for large counts.

No additional computational cost compared to existing methods.

Abstract

Variational Bayesian inference and (collapsed) Gibbs sampling are the two important classes of inference algorithms for Bayesian networks. Both have their advantages and disadvantages: collapsed Gibbs sampling is unbiased but is also inefficient for large count values and requires averaging over many samples to reduce variance. On the other hand, variational Bayesian inference is efficient and accurate for large count values but suffers from bias for small counts. We propose a hybrid algorithm that combines the best of both worlds: it samples very small counts and applies variational updates to large counts. This hybridization is shown to significantly improve testset perplexity relative to variational inference at no computational cost.