Cyclic Coordinate Update Algorithms for Fixed-Point Problems: Analysis and Applications

TL;DR

This paper proves convergence of cyclic coordinate-update algorithms for fixed-point problems with nonexpansive operators and demonstrates their efficiency in various applications like image reconstruction and matrix factorization.

Contribution

It establishes convergence results for cyclic coordinate-update algorithms in fixed-point problems and explores their practical advantages in optimization tasks.

Findings

Coordinate-update algorithms converge under proper step sizes.

Cyclic and shuffled cyclic rules outperform random selection in speed.

Algorithms significantly outperform standard fixed-point iteration in experiments.

Abstract

Many problems reduce to the fixed-point problem of solving $x=T(x)$. To this problem, we apply the coordinate-update algorithms, which update only one or a few components of $x$ at each step. When each update is cheap, these algorithms are faster than the full fixed-point iteration (which updates all the components). In this paper, we focus on the coordinate-update algorithms based on the cyclic selection rules, where the ordering of coordinates in each cycle is arbitrary. These algorithms are fast, but their convergence is unknown in the fixed-point setting. When $T$ is a nonexpansive operator and has a fixed point, we show that the sequence of coordinate-update iterates converges to a fixed point under proper step sizes. This result applies to the primal-dual coordinate-update algorithms, which have applications to optimization problems with nonseparable nonsmooth objectives, as well as global linear constraints. Numerically, we apply coordinate-update algorithms with the cyclic, shuffled cyclic, and random selection rules to $\ell_1$ robust least squares, a CT image reconstruction problem, as well as nonnegative matrix factorization. They converge much faster than the standard fixed-point iteration. Among the three rules, cyclic and shuffled cyclic rules are overall faster than the random rule.