Riemannian coordinate descent algorithms on matrix manifolds

TL;DR

This paper introduces coordinate descent algorithms tailored for optimization on matrix manifolds, enabling efficient updates by focusing on subsets of variables while maintaining manifold constraints, with proven convergence and practical applications.

Contribution

It develops a general framework for coordinate descent on various matrix manifolds, including new algorithms with low per-iteration cost and convergence analysis.

Findings

Algorithms are computationally efficient with low per-iteration cost.

Proposed methods converge under certain conditions.

Empirical results demonstrate effectiveness in multiple applications.

Abstract

Many machine learning applications are naturally formulated as optimization problems on Riemannian manifolds. The main idea behind Riemannian optimization is to maintain the feasibility of the variables while moving along a descent direction on the manifold. This results in updating all the variables at every iteration. In this work, we provide a general framework for developing computationally efficient coordinate descent (CD) algorithms on matrix manifolds that allows updating only a few variables at every iteration while adhering to the manifold constraint. In particular, we propose CD algorithms for various manifolds such as Stiefel, Grassmann, (generalized) hyperbolic, symplectic, and symmetric positive (semi)definite. While the cost per iteration of the proposed CD algorithms is low, we further develop a more efficient variant via a first-order approximation of the objective function. We analyze their convergence and complexity, and empirically illustrate their efficacy in several applications.