Asynchronous Distributed Semi-Stochastic Gradient Optimization

TL;DR

This paper introduces a distributed asynchronous SGD algorithm with variance reduction that converges linearly and outperforms existing methods in speed and solution quality on large-scale problems.

Contribution

It proposes a novel asynchronous distributed SGD algorithm that combines variance reduction with linear convergence guarantees, improving efficiency and solution quality.

Findings

Outperforms state-of-the-art algorithms in wall clock time

Achieves linear convergence with a constant learning rate

Demonstrates effectiveness on Google Cloud Platform

Abstract

With the recent proliferation of large-scale learning problems,there have been a lot of interest on distributed machine learning algorithms, particularly those that are based on stochastic gradient descent (SGD) and its variants. However, existing algorithms either suffer from slow convergence due to the inherent variance of stochastic gradients, or have a fast linear convergence rate but at the expense of poorer solution quality. In this paper, we combine their merits by proposing a fast distributed asynchronous SGD-based algorithm with variance reduction. A constant learning rate can be used, and it is also guaranteed to converge linearly to the optimal solution. Experiments on the Google Cloud Computing Platform demonstrate that the proposed algorithm outperforms state-of-the-art distributed asynchronous algorithms in terms of both wall clock time and solution quality.