Distributed stochastic optimization for deep learning (thesis)

TL;DR

This thesis introduces Elastic Averaging SGD (EASGD), a distributed stochastic optimization method for deep learning that improves training speed, accuracy, and communication efficiency, with theoretical analysis and empirical validation on large datasets.

Contribution

The paper proposes EASGD, analyzes its convergence and stability, and demonstrates its advantages over existing methods like DOWNPOUR, including scalability and reduced communication.

Findings

EASGD accelerates training and improves test accuracy.

EASGD requires less communication than baseline methods.

The spread of data impacts convergence and stability.

Abstract

We study the problem of how to distribute the training of large-scale deep learning models in the parallel computing environment. We propose a new distributed stochastic optimization method called Elastic Averaging SGD (EASGD). We analyze the convergence rate of the EASGD method in the synchronous scenario and compare its stability condition with the existing ADMM method in the round-robin scheme. An asynchronous and momentum variant of the EASGD method is applied to train deep convolutional neural networks for image classification on the CIFAR and ImageNet datasets. Our approach accelerates the training and furthermore achieves better test accuracy. It also requires a much smaller amount of communication than other common baseline approaches such as the DOWNPOUR method. We then investigate the limit in speedup of the initial and the asymptotic phase of the mini-batch SGD, the momentum SGD, and the EASGD methods. We find that the spread of the input data distribution has a big impact on their initial convergence rate and stability region. We also find a surprising connection between the momentum SGD and the EASGD method with a negative moving average rate. A non-convex case is also studied to understand when EASGD can get trapped by a saddle point. Finally, we scale up the EASGD method by using a tree structured network topology. We show empirically its advantage and challenge. We also establish a connection between the EASGD and the DOWNPOUR method with the classical Jacobi and the Gauss-Seidel method, thus unifying a class of distributed stochastic optimization methods.