Innovation Compression for Communication-efficient Distributed Optimization with Linear Convergence

TL;DR

This paper introduces the COLD and Dyna-COLD algorithms for communication-efficient distributed optimization, achieving linear convergence with compressed information, including for binary quantizers, and demonstrating improved performance over existing methods.

Contribution

The paper proposes the COLD and Dyna-COLD algorithms that enable linear convergence in distributed optimization with compressed communication, including for a broader class of compressors.

Findings

COLD achieves linear convergence with $ ext{delta}$-contracted compressors.

Dyna-COLD extends convergence guarantees to binary quantizers.

Numerical experiments confirm the algorithms' efficiency and advantages.

Abstract

Information compression is essential to reduce communication cost in distributed optimization over peer-to-peer networks. This paper proposes a communication-efficient linearly convergent distributed (COLD) algorithm to solve strongly convex optimization problems. By compressing innovation vectors, which are the differences between decision vectors and their estimates, COLD is able to achieve linear convergence for a class of $\delta$-contracted compressors. We explicitly quantify how the compression affects the convergence rate and show that COLD matches the same rate of its uncompressed version. To accommodate a wider class of compressors that includes the binary quantizer, we further design a novel dynamical scaling mechanism and obtain the linearly convergent Dyna-COLD. Importantly, our results strictly improve existing results for the quantized consensus problem. Numerical experiments demonstrate the advantages of both algorithms under different compressors.