A better convergence analysis of the block coordinate descent method for large scale machine learning

TL;DR

This paper provides a significantly improved convergence analysis for the block coordinate descent method applied to large-scale smooth convex optimization, introducing a new lower bound on its complexity.

Contribution

It introduces the lowest known lower bound on the information-based complexity of BCD, using the Performance Estimation Problem technique.

Findings

New lower bound is 16p^3 times smaller than previous bounds.

Numerical tests confirm the theoretical analysis.

Enhanced understanding of BCD convergence properties.

Abstract

This paper considers the problems of unconstrained minimization of large scale smooth convex functions having block-coordinate-wise Lipschitz continuous gradients. The block coordinate descent (BCD) method are among the first optimization schemes suggested for solving such problems \cite{nesterov2012efficiency}. We obtain a new lower (to our best knowledge the lowest currently) bound that is $16p^3$ times smaller than the best known on the information-based complexity of BCD method based on an effective technique called Performance Estimation Problem (PEP) proposed by Drori and Teboulle \cite{drori2012performance} recently for analyzing the performance of first-order black box optimization methods. Numerical test confirms our analysis.