Local convergence of alternating low-rank optimization methods with overrelaxation

TL;DR

This paper analyzes the local convergence of overrelaxed alternating optimization methods for low-rank matrix and tensor problems, providing theoretical insights into their acceleration and optimal parameters.

Contribution

It establishes the local convergence properties of overrelaxation in low-rank optimization and derives optimal relaxation parameters using a linearization approach.

Findings

Convergence can be accelerated with appropriate overrelaxation.

Optimal relaxation parameters depend on the convergence rate of standard methods.

The analysis applies to both matrix and tensor low-rank problems.

Abstract

The local convergence of alternating optimization methods with overrelaxation for low-rank matrix and tensor problems is established. The analysis is based on the linearization of the method which takes the form of an SOR iteration for a positive semidefinite Hessian and can be studied in the corresponding quotient geometry of equivalent low-rank representations. In the matrix case, the optimal relaxation parameter for accelerating the local convergence can be determined from the convergence rate of the standard method. This result relies on a version of Young's SOR theorem for positive semidefinite $2 \times 2$ block systems.