Towards Faster Decentralized Stochastic Optimization with Communication Compression

TL;DR

This paper introduces MoTEF, a new decentralized optimization algorithm that combines communication compression, Momentum Tracking, and Error Feedback to improve efficiency and scalability in federated learning.

Contribution

MoTEF is the first method to effectively integrate compression with momentum and error feedback, addressing scalability and heterogeneity issues in decentralized optimization.

Findings

MoTEF outperforms existing methods under data heterogeneity.

It achieves faster convergence with less communication.

Validated through extensive numerical experiments.

Abstract

Communication efficiency has garnered significant attention as it is considered the main bottleneck for large-scale decentralized Machine Learning applications in distributed and federated settings. In this regime, clients are restricted to transmitting small amounts of quantized information to their neighbors over a communication graph. Numerous endeavors have been made to address this challenging problem by developing algorithms with compressed communication for decentralized non-convex optimization problems. Despite considerable efforts, the current results suffer from various issues such as non-scalability with the number of clients, requirements for large batches, or bounded gradient assumption. In this paper, we introduce MoTEF, a novel approach that integrates communication compression with Momentum Tracking and Error Feedback. Our analysis demonstrates that MoTEF achieves most of the desired properties, and significantly outperforms existing methods under arbitrary data heterogeneity. We provide numerical experiments to validate our theoretical findings and confirm the practical superiority of MoTEF.