A Scalable Asynchronous Distributed Algorithm for Topic Modeling

TL;DR

This paper introduces F+Nomad LDA, a scalable asynchronous distributed algorithm for topic modeling that efficiently handles millions of documents and thousands of topics using a modified Fenwick tree and an innovative asynchronous framework.

Contribution

The paper presents a novel algorithm combining a modified Fenwick tree with an asynchronous distributed framework for scalable topic modeling on massive datasets.

Findings

Outperforms state-of-the-art methods on large-scale datasets

Handles thousands of topics efficiently in $O( ext{log} T)$ time

Effectively distributes computation across multiple machines

Abstract

Learning meaningful topic models with massive document collections which contain millions of documents and billions of tokens is challenging because of two reasons: First, one needs to deal with a large number of topics (typically in the order of thousands). Second, one needs a scalable and efficient way of distributing the computation across multiple machines. In this paper we present a novel algorithm F+Nomad LDA which simultaneously tackles both these problems. In order to handle large number of topics we use an appropriately modified Fenwick tree. This data structure allows us to sample from a multinomial distribution over $T$ items in $O(\log T)$ time. Moreover, when topic counts change the data structure can be updated in $O(\log T)$ time. In order to distribute the computation across multiple processor we present a novel asynchronous framework inspired by the Nomad algorithm of \cite{YunYuHsietal13}. We show that F+Nomad LDA significantly outperform state-of-the-art on massive problems which involve millions of documents, billions of words, and thousands of topics.