On Distributed Adaptive Optimization with Gradient Compression

TL;DR

This paper introduces COMP-AMS, a distributed adaptive optimization framework that uses gradient compression with error feedback, achieving communication efficiency without sacrificing convergence speed or accuracy.

Contribution

It presents a simple, effective distributed adaptive optimization method with gradient compression, maintaining convergence rates and test accuracy while reducing communication costs.

Findings

COMP-AMS achieves the same convergence rate as standard AMSGrad.

The method exhibits linear speedup with the number of workers.

Numerical experiments confirm reduced communication with maintained accuracy.

Abstract

We study COMP-AMS, a distributed optimization framework based on gradient averaging and adaptive AMSGrad algorithm. Gradient compression with error feedback is applied to reduce the communication cost in the gradient transmission process. Our convergence analysis of COMP-AMS shows that such compressed gradient averaging strategy yields same convergence rate as standard AMSGrad, and also exhibits the linear speedup effect w.r.t. the number of local workers. Compared with recently proposed protocols on distributed adaptive methods, COMP-AMS is simple and convenient. Numerical experiments are conducted to justify the theoretical findings, and demonstrate that the proposed method can achieve same test accuracy as the full-gradient AMSGrad with substantial communication savings. With its simplicity and efficiency, COMP-AMS can serve as a useful distributed training framework for adaptive gradient methods.