Alternating Minimization Methods for Strongly Convex Optimization

TL;DR

This paper analyzes alternating minimization methods for convex optimization, establishing linear convergence under relaxed conditions and proposing accelerated variants with improved rates, applicable to multi-block problems and certain linear systems.

Contribution

It introduces convergence guarantees for alternating minimization under Polyak-Lojasiewicz and strong convexity, and develops accelerated methods with better complexity bounds.

Findings

Linear convergence under Polyak-Lojasiewicz condition for two-block problems.

Accelerated alternating minimization with square root dependence on condition number.

Application to approximating non-negative solutions of linear systems using KL divergence.

Abstract

{We consider alternating minimization procedures for convex optimization problems with variable divided in many block, each block being amenable for minimization with respect to its variable with freezed other variables blocks. In the case of two blocks, we prove a linear convergence rate for alternating minimization procedure under Polyak-Lojasiewicz condition, which can be seen as a relaxation of the strong convexity assumption. Under strong convexity assumption in many-blocks setting we provide an accelerated alternating minimization procedure with linear rate depending on the square root of the condition number as opposed to condition number for the non-accelerated method. We also mention an approximating non-negative solution to a linear system of equations $Ax=y$ with alternating minimization of Kullback-Leibler (KL) divergence between $Ax$ and $y$.