Factorbird - a Parameter Server Approach to Distributed Matrix Factorization

TL;DR

Factorbird is a scalable, flexible parameter server system designed for large-scale matrix factorization using SGD, capable of handling massive matrices exceeding memory limits and adaptable to various models and streaming data.

Contribution

Introduces Factorbird, a novel distributed matrix factorization system that scales to billions of non-zero entries and supports diverse models and streaming scenarios.

Findings

Successfully factorized a 38-billion non-zero matrix from Twitter data.

Demonstrated scalability and efficiency of the system on extremely large matrices.

Achieved state-of-the-art results in large-scale matrix factorization literature.

Abstract

We present Factorbird, a prototype of a parameter server approach for factorizing large matrices with Stochastic Gradient Descent-based algorithms. We designed Factorbird to meet the following desiderata: (a) scalability to tall and wide matrices with dozens of billions of non-zeros, (b) extensibility to different kinds of models and loss functions as long as they can be optimized using Stochastic Gradient Descent (SGD), and (c) adaptability to both batch and streaming scenarios. Factorbird uses a parameter server in order to scale to models that exceed the memory of an individual machine, and employs lock-free Hogwild!-style learning with a special partitioning scheme to drastically reduce conflicting updates. We also discuss other aspects of the design of our system such as how to efficiently grid search for hyperparameters at scale. We present experiments of Factorbird on a matrix built from a subset of Twitter's interaction graph, consisting of more than 38 billion non-zeros and about 200 million rows and columns, which is to the best of our knowledge the largest matrix on which factorization results have been reported in the literature.