Communication Compression for Decentralized Learning with Operator Splitting Methods

TL;DR

This paper introduces a novel communication compression framework for operator splitting methods in decentralized learning, significantly reducing communication costs and improving robustness to data heterogeneity.

Contribution

The paper proposes C-ECL, a new framework that compresses dual variable updates in ECL, enhancing efficiency and robustness compared to existing Gossip-based algorithms.

Findings

C-ECL achieves similar performance with fewer parameter exchanges.

C-ECL is more robust to data heterogeneity.

Experimental results validate the effectiveness of C-ECL.

Abstract

In decentralized learning, operator splitting methods using a primal-dual formulation (e.g., the Edge-Consensus Learning (ECL)) has been shown to be robust to heterogeneous data and has attracted significant attention in recent years. However, in the ECL, a node needs to exchange dual variables with its neighbors. These exchanges incur significant communication costs. For the Gossip-based algorithms, many compression methods have been proposed, but these Gossip-based algorithm do not perform well when the data distribution held by each node is statistically heterogeneous. In this work, we propose the novel framework of the compression methods for the ECL, called the Communication Compressed ECL (C-ECL). Specifically, we reformulate the update formulas of the ECL, and propose to compress the update values of the dual variables. We demonstrate experimentally that the C-ECL can achieve a nearly equivalent performance with fewer parameter exchanges than the ECL. Moreover, we demonstrate that the C-ECL is more robust to heterogeneous data than the Gossip-based algorithms.