Distributed and Stochastic Optimization Methods with Gradient Compression and Local Steps

TL;DR

This thesis introduces new theoretical frameworks and over 20 optimization methods for distributed stochastic optimization, including linearly converging error-compensated SGD and heterogeneous local-SGD, with improved complexity results.

Contribution

Develops novel theoretical frameworks and over 20 new optimization algorithms, including the first linearly converging error-compensated SGD and heterogeneous local-SGD.

Findings

First linearly converging error-compensated SGD

First linearly converging local-SGD for heterogeneous functions

Distributed methods with improved complexity bounds

Abstract

In this thesis, we propose new theoretical frameworks for the analysis of stochastic and distributed methods with error compensation and local updates. Using these frameworks, we develop more than 20 new optimization methods, including the first linearly converging Error-Compensated SGD and the first linearly converging Local-SGD for arbitrarily heterogeneous local functions. Moreover, the thesis contains several new distributed methods with unbiased compression for distributed non-convex optimization problems. The derived complexity results for these methods outperform the previous best-known results for the considered problems. Finally, we propose a new scalable decentralized fault-tolerant distributed method, and under reasonable assumptions, we derive the iteration complexity bounds for this method that match the ones of centralized Local-SGD.