Local AdaAlter: Communication-Efficient Stochastic Gradient Descent with Adaptive Learning Rates

TL;DR

This paper introduces Local AdaAlter, a communication-efficient stochastic gradient descent method with adaptive learning rates, proven to converge for non-convex problems and capable of reducing training time significantly.

Contribution

It presents a novel SGD variant that reduces communication overhead and adapts learning rates, with proven convergence for smooth non-convex problems.

Findings

Reduces communication overhead in distributed training

Achieves up to 30% reduction in training time on large datasets

Proven convergence for smooth non-convex optimization

Abstract

When scaling distributed training, the communication overhead is often the bottleneck. In this paper, we propose a novel SGD variant with reduced communication and adaptive learning rates. We prove the convergence of the proposed algorithm for smooth but non-convex problems. Empirical results show that the proposed algorithm significantly reduces the communication overhead, which, in turn, reduces the training time by up to 30% for the 1B word dataset.