On the Complexity of Parallel Coordinate Descent

TL;DR

This paper advances the understanding of parallel coordinate descent methods by providing sharper iteration complexity results, confirming monotonicity in expectation, and establishing high probability bounds for unbounded initial conditions.

Contribution

It offers improved theoretical analysis of PCDM, including new complexity bounds, monotonicity proof, and high probability results for unbounded initial levelsets.

Findings

Sharper iteration complexity bounds for PCDM

Proof of monotonicity in expectation

First high probability iteration complexity result for unbounded initial levelsets

Abstract

In this work we study the parallel coordinate descent method (PCDM) proposed by Richt\'arik and Tak\'a\v{c} [26] for minimizing a regularized convex function. We adopt elements from the work of Xiao and Lu [39], and combine them with several new insights, to obtain sharper iteration complexity results for PCDM than those presented in [26]. Moreover, we show that PCDM is monotonic in expectation, which was not confirmed in [26], and we also derive the first high probability iteration complexity result where the initial levelset is unbounded.