Accelerated Methods with Compressed Communications for Distributed Optimization Problems under Data Similarity

TL;DR

This paper introduces accelerated distributed optimization algorithms that combine data similarity, compression, and variance reduction techniques to mitigate communication bottlenecks in large-scale machine learning.

Contribution

It presents the first theoretically grounded accelerated algorithms that integrate unbiased and biased compression with data similarity, enhancing efficiency in distributed learning.

Findings

Algorithms achieve record convergence speeds.

Experimental results confirm improved efficiency across datasets.

Effective handling of both unbiased and biased compression methods.

Abstract

In recent years, as data and problem sizes have increased, distributed learning has become an essential tool for training high-performance models. However, the communication bottleneck, especially for high-dimensional data, is a challenge. Several techniques have been developed to overcome this problem. These include communication compression and implementation of local steps, which work particularly well when there is similarity of local data samples. In this paper, we study the synergy of these approaches for efficient distributed optimization. We propose the first theoretically grounded accelerated algorithms utilizing unbiased and biased compression under data similarity, leveraging variance reduction and error feedback frameworks. Our results are of record and confirmed by experiments on different average losses and datasets.