An Asynchronous Parallel Stochastic Coordinate Descent Algorithm

TL;DR

This paper introduces an asynchronous parallel stochastic coordinate descent algorithm that efficiently minimizes smooth functions, achieving linear convergence under strong convexity and near-linear speedup on multicore systems.

Contribution

It presents a novel asynchronous parallel stochastic coordinate descent method with proven convergence rates and demonstrates its effectiveness on multicore hardware.

Findings

Achieves linear convergence on strongly convex functions.

Attains sublinear ($1/K$) convergence on general convex functions.

Demonstrates near-linear speedup on 40-core systems.

Abstract

We describe an asynchronous parallel stochastic coordinate descent algorithm for minimizing smooth unconstrained or separably constrained functions. The method achieves a linear convergence rate on functions that satisfy an essential strong convexity property and a sublinear rate ($1/K$) on general convex functions. Near-linear speedup on a multicore system can be expected if the number of processors is $O(n^{1/2})$ in unconstrained optimization and $O(n^{1/4})$ in the separable-constrained case, where $n$ is the number of variables. We describe results from implementation on 40-core processors.