Markovian Compression: Looking to the Past Helps Accelerate the Future

TL;DR

This paper introduces Markovian compressors for distributed optimization, improving convergence and efficiency in communication-constrained settings by leveraging past information, with theoretical guarantees and empirical validation.

Contribution

It proposes a novel family of Markovian compressors integrated into QSGD and momentum QSGD, with convergence analysis for various convexity conditions.

Findings

Markovian compressors outperform existing schemes in experiments.

The algorithms converge under non-convex and convex conditions.

Practical improvements observed on CIFAR-10 and GLUE datasets.

Abstract

This paper deals with distributed optimization problems that use compressed communication to achieve efficient performance and mitigate communication bottleneck. We propose a family of compression schemes in which operators transform vectors fed to their input according to a Markov chain, i.e. the stochasticity of the compressors depends on previous iterations. The compressors are implemented in the vanilla Quantized Stochastic Gradient Descent algorithm (QSGD), and, to further improve the efficiency and convergence rate, in the momentum accelerated QSGD. We provide convergence results for our algorithms with Markovian compressors, the analysis covers non-convex, Polyak-Lojasiewicz, and strongly convex cases. To demonstrate the applicability of our approach to distributed data-parallel optimization problems, we conduct experiments on the CIFAR-10 and GLUE datasets with the Resnet-18 and DeBERTaV3 models. Practical results show the superiority of methods that use our compressor design over existing schemes.