Alternating least squares as moving subspace correction

TL;DR

This paper offers a new perspective on the convergence of alternating optimization for low-rank matrices and tensors, linking it to subspace correction methods and providing insights into convergence conditions and rates.

Contribution

It introduces an abstract interpretation of convergence as moving subspace correction, connecting classical methods with low-rank manifold curvature, and derives the asymptotic convergence rate for a key numerical method.

Findings

Reformulation of convergence conditions using subspace correction perspective

Derivation of the asymptotic convergence rate for the two-sided block power method

Numerical experiments validating theoretical insights

Abstract

In this note we take a new look at the local convergence of alternating optimization methods for low-rank matrices and tensors. Our abstract interpretation as sequential optimization on moving subspaces yields insightful reformulations of some known convergence conditions that focus on the interplay between the contractivity of classical multiplicative Schwarz methods with overlapping subspaces and the curvature of low-rank matrix and tensor manifolds. While the verification of the abstract conditions in concrete scenarios remains open in most cases, we are able to provide an alternative and conceptually simple derivation of the asymptotic convergence rate of the two-sided block power method of numerical algebra for computing the dominant singular subspaces of a rectangular matrix. This method is equivalent to an alternating least squares method applied to a distance function. The theoretical results are illustrated and validated by numerical experiments.