EControl: Fast Distributed Optimization with Compression and Error Control

TL;DR

EControl introduces a novel error compensation mechanism for distributed optimization with compression, achieving fast convergence without restrictive assumptions in heterogeneous data settings.

Contribution

The paper proposes EControl, a new method that effectively regulates error feedback in compressed distributed optimization, ensuring stability and fast convergence.

Findings

EControl converges rapidly in convex and nonconvex problems.

It works effectively without assumptions like bounded gradients or data heterogeneity.

Numerical results confirm the theoretical advantages of EControl.

Abstract

Modern distributed training relies heavily on communication compression to reduce the communication overhead. In this work, we study algorithms employing a popular class of contractive compressors in order to reduce communication overhead. However, the naive implementation often leads to unstable convergence or even exponential divergence due to the compression bias. Error Compensation (EC) is an extremely popular mechanism to mitigate the aforementioned issues during the training of models enhanced by contractive compression operators. Compared to the effectiveness of EC in the data homogeneous regime, the understanding of the practicality and theoretical foundations of EC in the data heterogeneous regime is limited. Existing convergence analyses typically rely on strong assumptions such as bounded gradients, bounded data heterogeneity, or large batch accesses, which are often infeasible in modern machine learning applications. We resolve the majority of current issues by proposing EControl, a novel mechanism that can regulate error compensation by controlling the strength of the feedback signal. We prove fast convergence for EControl in standard strongly convex, general convex, and nonconvex settings without any additional assumptions on the problem or data heterogeneity. We conduct extensive numerical evaluations to illustrate the efficacy of our method and support our theoretical findings.