Decentralised convex optimisation with probability-proportional-to-size quantization

TL;DR

This paper introduces a novel probability-proportional-to-size quantization method for distributed convex optimization, improving communication efficiency and convergence in decentralized settings, with applications to Wasserstein barycenters.

Contribution

It proposes a new quantization technique and accelerated stochastic gradient methods tailored for affine-constrained convex problems in decentralized optimization.

Findings

Achieves convergence rates with bounds on large deviations

Demonstrates effectiveness in decentralized Wasserstein barycenter computation

Reduces communication costs in distributed optimization

Abstract

Communication is one of the bottlenecks of distributed optimisation and learning. To overcome this bottleneck, we propose a novel quantization method that transforms a vector into a sample of components' indices drawn from a categorical distribution with probabilities proportional to values at those components. Then, we propose a primal and a primal-dual accelerated stochastic gradient methods that use our proposed quantization, and derive their convergence rates in terms of probabilities of large deviations. We focus on affine-constrained convex optimisation and its application to decentralised distributed optimisation problems. To illustrate the work of our algorithm, we apply it to the decentralised computation of semi-discrete entropy regularized Wasserstein barycenters.