Fast Asynchronous Parallel Stochastic Gradient Decent

TL;DR

This paper introduces AsySVRG, a fast asynchronous parallel stochastic gradient descent method that improves convergence and efficiency over existing methods like Hogwild! for large-scale machine learning tasks.

Contribution

The paper proposes AsySVRG, an asynchronous parallel SGD algorithm that combines SVRG with an innovative asynchronous strategy, enhancing performance in large-scale settings.

Findings

AsySVRG outperforms Hogwild! in convergence rate.

AsySVRG reduces computation cost compared to existing methods.

Theoretical analysis confirms faster convergence of AsySVRG.

Abstract

Stochastic gradient descent~(SGD) and its variants have become more and more popular in machine learning due to their efficiency and effectiveness. To handle large-scale problems, researchers have recently proposed several parallel SGD methods for multicore systems. However, existing parallel SGD methods cannot achieve satisfactory performance in real applications. In this paper, we propose a fast asynchronous parallel SGD method, called AsySVRG, by designing an asynchronous strategy to parallelize the recently proposed SGD variant called stochastic variance reduced gradient~(SVRG). Both theoretical and empirical results show that AsySVRG can outperform existing state-of-the-art parallel SGD methods like Hogwild! in terms of convergence rate and computation cost.