Partitioning Algorithms for Improving Efficiency of Topic Modeling Parallelization

TL;DR

This paper introduces three novel partitioning algorithms that enhance the efficiency of parallelized topic modeling, significantly reducing overhead and improving load balancing, applicable to models like LDA and Bag of Timestamps.

Contribution

The paper proposes three new partitioning algorithms that outperform existing methods in speed and load balancing for parallel topic modeling, with demonstrated extensibility to related models.

Findings

Algorithms run faster than existing methods.

Improved load balancing in parallel topic modeling.

Validated on datasets including NIPS, NYTimes, and a large scientific publication corpus.

Abstract

Topic modeling is a very powerful technique in data analysis and data mining but it is generally slow. Many parallelization approaches have been proposed to speed up the learning process. However, they are usually not very efficient because of the many kinds of overhead, especially the load-balancing problem. We address this problem by proposing three partitioning algorithms, which either run more quickly or achieve better load balance than current partitioning algorithms. These algorithms can easily be extended to improve parallelization efficiency on other topic models similar to LDA, e.g., Bag of Timestamps, which is an extension of LDA with time information. We evaluate these algorithms on two popular datasets, NIPS and NYTimes. We also build a dataset containing over 1,000,000 scientific publications in the computer science domain from 1951 to 2010 to experiment with Bag of Timestamps parallelization, which we design to demonstrate the proposed algorithms' extensibility. The results strongly confirm the advantages of these algorithms.