High-Performance Distributed ML at Scale through Parameter Server Consistency Models

TL;DR

This paper analyzes the theoretical and empirical aspects of consistency models in distributed Parameter Server frameworks for Machine Learning, proposing improvements to accelerate convergence and ensure correctness.

Contribution

It provides a theoretical study of consistency models in PS systems, introduces an improved eager communication mechanism, and implements a new system to enhance ML training speed.

Findings

Certain consistency models guarantee correct ML outputs.

The eager communication mechanism improves convergence speed.

The new PS system outperforms existing frameworks in training time.

Abstract

As Machine Learning (ML) applications increase in data size and model complexity, practitioners turn to distributed clusters to satisfy the increased computational and memory demands. Unfortunately, effective use of clusters for ML requires considerable expertise in writing distributed code, while highly-abstracted frameworks like Hadoop have not, in practice, approached the performance seen in specialized ML implementations. The recent Parameter Server (PS) paradigm is a middle ground between these extremes, allowing easy conversion of single-machine parallel ML applications into distributed ones, while maintaining high throughput through relaxed "consistency models" that allow inconsistent parameter reads. However, due to insufficient theoretical study, it is not clear which of these consistency models can really ensure correct ML algorithm output; at the same time, there remain many theoretically-motivated but undiscovered opportunities to maximize computational throughput. Motivated by this challenge, we study both the theoretical guarantees and empirical behavior of iterative-convergent ML algorithms in existing PS consistency models. We then use the gleaned insights to improve a consistency model using an "eager" PS communication mechanism, and implement it as a new PS system that enables ML algorithms to reach their solution more quickly.