Inexact Coordinate Descent: Complexity and Preconditioning

TL;DR

This paper introduces an inexact coordinate descent method for convex optimization that relaxes the need for exact subproblem solutions, incorporating preconditioning and iterative techniques for improved efficiency.

Contribution

It proposes a novel inexact block coordinate descent algorithm with theoretical guarantees, extending existing methods to allow inexact updates and practical acceleration techniques.

Findings

The method achieves comparable convergence guarantees to exact methods.

Preconditioning significantly accelerates convergence.

Inexact updates reduce computational complexity.

Abstract

In this paper we consider the problem of minimizing a convex function using a randomized block coordinate descent method. One of the key steps at each iteration of the algorithm is determining the update to a block of variables. Existing algorithms assume that in order to compute the update, a particular subproblem is solved exactly. In his work we relax this requirement, and allow for the subproblem to be solved inexactly, leading to an inexact block coordinate descent method. Our approach incorporates the best known results for exact updates as a special case. Moreover, these theoretical guarantees are complemented by practical considerations: the use of iterative techniques to determine the update as well as the use of preconditioning for further acceleration.