On the Worst-Case Analysis of Cyclic Block Coordinate Descent type Algorithms

TL;DR

This paper analyzes the worst-case convergence behavior of various cyclic block coordinate descent algorithms, introduces improved bounds using the PEP approach, and uncovers phenomena like scale-invariance and performance bounds.

Contribution

It extends the PEP framework to convex coordinate-wise smooth functions, providing tighter bounds and new insights into the behavior of BCD algorithms.

Findings

Improved numerical upper bounds on convergence rates for CCD, AM, and CACD.

Demonstrates convergence of CCD under more natural assumptions.

Identifies scale-invariance and lower bounds related to full gradient descent.

Abstract

We study the worst-case behavior of Block Coordinate Descent (BCD) type algorithms for unconstrained minimization of coordinate-wise smooth convex functions. This behavior is indeed not completely understood, and the practical success of these algorithms is not fully explained by current convergence analyses. We extend the recently proposed Performance Estimation Problem (PEP) approach to convex coordinate-wise smooth functions by proposing necessary interpolation conditions. We then exploit this to obtain improved numerical upper bounds on the worst-case convergence rate of three different BCD algorithms, namely Cyclic Coordinate Descent (CCD), Alternating Minimization (AM), and a Cyclic version of the Random Accelerated Coordinate Descent introduced in Fercoq and Richt\'arik (2015) (CACD), substantially outperforming the best current bounds in some situations. In addition, we show the convergence of the CCD algorithm with more natural assumptions in the context of convex optimization than those typically made in the literature. Our methodology uncovers a number of phenomena, some of which can be formally established. These include a scale-invariance property of the worst case of CCD with respect to the coordinate-wise smoothness constants and a lower bound on the worst-case performance of CCD which is equal to the number of blocks times the worst-case of full gradient descent over the class of smooth convex functions. We also adapt our framework to the analysis of random BCD algorithms, and present numerical results showing that the standard acceleration scheme in Fercoq and Richt\'arik (2015) appears to be inefficient for deterministic algorithms.