Fast Distributed Coordinate Descent for Non-Strongly Convex Losses

TL;DR

This paper introduces a fast distributed coordinate descent algorithm for large-scale non-strongly convex optimization, achieving optimal convergence rates and demonstrating scalability on supercomputers for problems with billions of variables.

Contribution

It presents a novel distributed randomized coordinate descent method with proven optimal convergence for non-strongly convex functions, and demonstrates its scalability on supercomputers.

Findings

Achieves $O(1/k^2)$ convergence rate for non-strongly convex losses.

Successfully solves a synthetic LASSO problem with 50 billion variables.

Implemented on the UK's largest supercomputer, showing high scalability.

Abstract

We propose an efficient distributed randomized coordinate descent method for minimizing regularized non-strongly convex loss functions. The method attains the optimal $O(1/k^2)$ convergence rate, where $k$ is the iteration counter. The core of the work is the theoretical study of stepsize parameters. We have implemented the method on Archer - the largest supercomputer in the UK - and show that the method is capable of solving a (synthetic) LASSO optimization problem with 50 billion variables.