Even Faster Accelerated Coordinate Descent Using Non-Uniform Sampling

TL;DR

This paper introduces a novel non-uniform sampling method for accelerated coordinate descent, significantly improving its speed by up to a factor of  in large-scale optimization tasks like machine learning and linear systems.

Contribution

It presents a new non-uniform sampling scheme based on coordinate smoothness, achieving faster convergence rates for accelerated coordinate descent algorithms.

Findings

Achieves up to  speed-up in running time.

Effective in empirical risk minimization and linear system solving.

Validates improvements through theoretical analysis and practical experiments.

Abstract

Accelerated coordinate descent is widely used in optimization due to its cheap per-iteration cost and scalability to large-scale problems. Up to a primal-dual transformation, it is also the same as accelerated stochastic gradient descent that is one of the central methods used in machine learning. In this paper, we improve the best known running time of accelerated coordinate descent by a factor up to $\sqrt{n}$. Our improvement is based on a clean, novel non-uniform sampling that selects each coordinate with a probability proportional to the square root of its smoothness parameter. Our proof technique also deviates from the classical estimation sequence technique used in prior work. Our speed-up applies to important problems such as empirical risk minimization and solving linear systems, both in theory and in practice.