Towards Understanding Acceleration Tradeoff between Momentum and Asynchrony in Nonconvex Stochastic Optimization

TL;DR

This paper analyzes the convergence tradeoff between asynchrony and momentum in Async-MSGD for nonconvex stochastic optimization, using streaming PCA as a case study, and reveals a fundamental balance needed for convergence and acceleration.

Contribution

It provides the first theoretical analysis of Async-MSGD's convergence for nonconvex problems, highlighting the tradeoff between asynchrony and momentum.

Findings

Asynchrony and momentum have a fundamental tradeoff affecting convergence.

Reducing momentum is necessary to leverage asynchrony for acceleration.

Experimental results support the theoretical tradeoff in deep neural network training.

Abstract

Asynchronous momentum stochastic gradient descent algorithms (Async-MSGD) is one of the most popular algorithms in distributed machine learning. However, its convergence properties for these complicated nonconvex problems is still largely unknown, because of the current technical limit. Therefore, in this paper, we propose to analyze the algorithm through a simpler but nontrivial nonconvex problem - streaming PCA, which helps us to understand Aync-MSGD better even for more general problems. Specifically, we establish the asymptotic rate of convergence of Async-MSGD for streaming PCA by diffusion approximation. Our results indicate a fundamental tradeoff between asynchrony and momentum: To ensure convergence and acceleration through asynchrony, we have to reduce the momentum (compared with Sync-MSGD). To the best of our knowledge, this is the first theoretical attempt on understanding Async-MSGD for distributed nonconvex stochastic optimization. Numerical experiments on both streaming PCA and training deep neural networks are provided to support our findings for Async-MSGD.