Probabilistic Synchronous Parallel

TL;DR

This paper introduces Probabilistic Synchronous Parallel (PSP), a novel barrier control method for distributed learning that enhances convergence speed and scalability of SGD by incorporating sampling, with theoretical analysis and practical implementation.

Contribution

The paper proposes PSP, a new barrier control technique that improves distributed SGD performance by integrating sampling, and demonstrates its effectiveness through theoretical analysis and a full-featured framework.

Findings

PSP improves convergence speed over BSP, SSP, and ASP.

PSP enhances scalability in distributed SGD.

Theoretical convergence guarantees are established for PSP-based SGD.

Abstract

Most machine learning and deep neural network algorithms rely on certain iterative algorithms to optimise their utility/cost functions, e.g. Stochastic Gradient Descent. In distributed learning, the networked nodes have to work collaboratively to update the model parameters, and the way how they proceed is referred to as synchronous parallel design (or barrier control). Synchronous parallel protocol is the building block of any distributed learning framework, and its design has direct impact on the performance and scalability of the system. In this paper, we propose a new barrier control technique - Probabilistic Synchronous Parallel (PSP). Com- paring to the previous Bulk Synchronous Parallel (BSP), Stale Synchronous Parallel (SSP), and (Asynchronous Parallel) ASP, the proposed solution e ectively improves both the convergence speed and the scalability of the SGD algorithm by introducing a sampling primitive into the system. Moreover, we also show that the sampling primitive can be applied atop of the existing barrier control mechanisms to derive fully distributed PSP-based synchronous parallel. We not only provide a thorough theoretical analysis1 on the convergence of PSP-based SGD algorithm, but also implement a full-featured distributed learning framework called Actor and perform intensive evaluation atop of it.