Anderson acceleration of coordinate descent

TL;DR

This paper introduces an extrapolation-based acceleration method for coordinate descent, demonstrating significant practical speed-ups over existing inertial and gradient-based methods across various machine learning tasks.

Contribution

It proposes a novel accelerated coordinate descent method using extrapolation, improving practical performance over prior inertial and gradient extrapolation techniques.

Findings

Significant speed-up in coordinate descent for multiple problems

Outperforms inertial accelerated coordinate descent in experiments

Effective on least squares, Lasso, elastic net, and logistic regression

Abstract

Acceleration of first order methods is mainly obtained via inertial techniques \`a la Nesterov, or via nonlinear extrapolation. The latter has known a recent surge of interest, with successful applications to gradient and proximal gradient techniques. On multiple Machine Learning problems, coordinate descent achieves performance significantly superior to full-gradient methods. Speeding up coordinate descent in practice is not easy: inertially accelerated versions of coordinate descent are theoretically accelerated, but might not always lead to practical speed-ups. We propose an accelerated version of coordinate descent using extrapolation, showing considerable speed up in practice, compared to inertial accelerated coordinate descent and extrapolated (proximal) gradient descent. Experiments on least squares, Lasso, elastic net and logistic regression validate the approach.