Parallel Stochastic Asynchronous Coordinate Descent: Tight Bounds on the Possible Parallelism

TL;DR

This paper establishes tight bounds on the maximum parallelism achievable in asynchronous stochastic coordinate descent algorithms, confirming the theoretical limits of linear speedup in parallel optimization.

Contribution

It proves that the known bounds on parallelism are tight for nearly all parameter values, providing a precise characterization of the algorithm's scalability.

Findings

The established bounds are tight for almost all parameter values.

Linear speedup is limited by the ratio of Lipschitz parameters.

The results confirm the optimality of existing parallel coordinate descent methods.

Abstract

Several works have shown linear speedup is achieved by an asynchronous parallel implementation of stochastic coordinate descent so long as there is not too much parallelism. More specifically, it is known that if all updates are of similar duration, then linear speedup is possible with up to $\Theta(\sqrt n L_{\max}/L_{\overline{\mathrm{res}}})$ processors, where $L_{\max}$ and $L_{\overline{\mathrm{res}}}$ are suitable Lipschitz parameters. This paper shows the bound is tight for almost all possible values of these parameters.