High Performance Latent Variable Models

TL;DR

This paper presents a highly scalable and efficient latent variable modeling system capable of handling industry-scale data with improved robustness and accuracy, utilizing advanced distributed inference techniques and sophisticated statistical models.

Contribution

The paper introduces a system that scales latent variable models to hundreds of billions of tokens using novel distributed inference and advanced statistical modeling beyond LDA.

Findings

Operates at a scale hundreds of billions of tokens

Achieves faster performance than previous state-of-the-art systems

Demonstrates robustness and accuracy at large scale

Abstract

Latent variable models have accumulated a considerable amount of interest from the industry and academia for their versatility in a wide range of applications. A large amount of effort has been made to develop systems that is able to extend the systems to a large scale, in the hope to make use of them on industry scale data. In this paper, we describe a system that operates at a scale orders of magnitude higher than previous works, and an order of magnitude faster than state-of-the-art system at the same scale, at the same time showing more robustness and more accurate results. Our system uses a number of advances in distributed inference: high performance in synchronization of sufficient statistics with relaxed consistency model; fast sampling, using the Metropolis-Hastings-Walker method to overcome dense generative models; statistical modeling, moving beyond Latent Dirichlet Allocation (LDA) to Pitman-Yor distributions (PDP) and Hierarchical Dirichlet Process (HDP) models; sophisticated parameter projection schemes, to resolve the conflicts within the constraint between parameters arising from the relaxed consistency model. This work significantly extends the domain of applicability of what is commonly known as the Parameter Server. We obtain results with up to hundreds billion oftokens, thousands of topics, and a vocabulary of a few million token-types, using up to 60,000 processor cores operating on a production cluster of a large Internet company. This demonstrates the feasibility to scale to problems orders of magnitude larger than any previously published work.