Balancing the Communication Load of Asynchronously Parallelized Machine Learning Algorithms

TL;DR

This paper explores how asynchronous communication affects the performance of parallelized SGD in large-scale machine learning, introducing a novel algorithm to balance communication load adaptively.

Contribution

It presents a new algorithm for automatic balancing of asynchronous communication load in ASGD, improving scalability and efficiency in variable network conditions.

Findings

ASGD outperforms traditional parallel algorithms for large-scale ML.

The proposed load balancing algorithm adapts to changing network bandwidths.

Performance improvements are demonstrated on HTC cluster and cloud environments.

Abstract

Stochastic Gradient Descent (SGD) is the standard numerical method used to solve the core optimization problem for the vast majority of machine learning (ML) algorithms. In the context of large scale learning, as utilized by many Big Data applications, efficient parallelization of SGD is in the focus of active research. Recently, we were able to show that the asynchronous communication paradigm can be applied to achieve a fast and scalable parallelization of SGD. Asynchronous Stochastic Gradient Descent (ASGD) outperforms other, mostly MapReduce based, parallel algorithms solving large scale machine learning problems. In this paper, we investigate the impact of asynchronous communication frequency and message size on the performance of ASGD applied to large scale ML on HTC cluster and cloud environments. We introduce a novel algorithm for the automatic balancing of the asynchronous communication load, which allows to adapt ASGD to changing network bandwidths and latencies.