Communication-Efficient Network-Distributed Optimization with Differential-Coded Compressors

TL;DR

This paper introduces a differential-coded compressed decentralized gradient descent (DC-DGD) algorithm that improves communication efficiency in network-distributed optimization, works with general SNR-constrained compressors, and maintains convergence rates.

Contribution

The paper proposes a novel DC-DGD algorithm that relaxes noise assumptions, maintains convergence rates, and reduces complexity, along with a hybrid compression scheme for minimal communication cost.

Findings

DC-DGD works with general SNR-constrained compressors.

The convergence rate of DC-DGD matches that of classic DGD.

Experimental results confirm the efficiency and effectiveness of DC-DGD and the hybrid scheme.

Abstract

Network-distributed optimization has attracted significant attention in recent years due to its ever-increasing applications. However, the classic decentralized gradient descent (DGD) algorithm is communication-inefficient for large-scale and high-dimensional network-distributed optimization problems. To address this challenge, many compressed DGD-based algorithms have been proposed. However, most of the existing works have high complexity and assume compressors with bounded noise power. To overcome these limitations, in this paper, we propose a new differential-coded compressed DGD (DC-DGD) algorithm. The key features of DC-DGD include: i) DC-DGD works with general SNR-constrained compressors, relaxing the bounded noise power assumption; ii) The differential-coded design entails the same convergence rate as the original DGD algorithm; and iii) DC-DGD has the same low-complexity structure as the original DGD due to a {\em self-noise-reduction effect}. Moreover, the above features inspire us to develop a hybrid compression scheme that offers a systematic mechanism to minimize the communication cost. Finally, we conduct extensive experiments to verify the efficacy of the proposed DC-DGD and hybrid compressor.