CompressedScaffnew: The First Theoretical Double Acceleration of Communication from Local Training and Compression in Distributed Optimization

TL;DR

CompressedScaffnew is a novel distributed optimization algorithm that combines local training and compression to achieve the first double acceleration, converging linearly in strongly convex settings.

Contribution

It introduces the first method to jointly accelerate distributed optimization through local training and compression, improving convergence rates.

Findings

Achieves linear convergence to the exact solution in strongly convex problems.

Provides a doubly accelerated rate benefiting from local training and compression.

Improves dependency on condition number and model dimension.

Abstract

In distributed optimization, a large number of machines alternate between local computations and communication with a coordinating server. Communication, which can be slow and costly, is the main bottleneck in this setting. To reduce this burden and therefore accelerate distributed gradient descent, two strategies are popular: 1) communicate less frequently; that is, perform several iterations of local computations between the communication rounds; and 2) communicate compressed information instead of full-dimensional vectors. We propose CompressedScaffnew, the first algorithm for distributed optimization that jointly harnesses these two strategies and converges linearly to an exact solution in the strongly convex setting, with a doubly accelerated rate: it benefits from the two acceleration mechanisms provided by local training and compression, namely a better dependency on the condition number of the functions and on the dimension of the model, respectively.