Inference in topic models: sparsity and trade-off

TL;DR

This paper explores the use of the Frank-Wolfe algorithm for efficient, sparse posterior inference in topic models, significantly speeding up LDA learning on large-scale and streaming data.

Contribution

It introduces the ML-FW method utilizing Frank-Wolfe for fast, scalable LDA inference, demonstrating substantial speed improvements over existing methods.

Findings

ML-FW is tens to thousands of times faster than state-of-the-art methods.

The approach effectively handles massive and streaming data.

FW provides beneficial properties for sparse posterior inference.

Abstract

Topic models are popular for modeling discrete data (e.g., texts, images, videos, links), and provide an efficient way to discover hidden structures/semantics in massive data. One of the core problems in this field is the posterior inference for individual data instances. This problem is particularly important in streaming environments, but is often intractable. In this paper, we investigate the use of the Frank-Wolfe algorithm (FW) for recovering sparse solutions to posterior inference. From detailed elucidation of both theoretical and practical aspects, FW exhibits many interesting properties which are beneficial to topic modeling. We then employ FW to design fast methods, including ML-FW, for learning latent Dirichlet allocation (LDA) at large scales. Extensive experiments show that to reach the same predictiveness level, ML-FW can perform tens to thousand times faster than existing state-of-the-art methods for learning LDA from massive/streaming data.