When geometry meets optimization theory: partially orthogonal tensors

TL;DR

This paper introduces a block coordinate descent algorithm for low rank partially orthogonal tensor approximation, providing the first global convergence guarantee and explicit convergence rates, including linear convergence for generic tensors.

Contribution

It develops a novel algorithm with proven global convergence and explicit rates for low rank partially orthogonal tensor approximation, filling a gap in theoretical guarantees.

Findings

Algorithm converges globally to a KKT point.

Achieves sublinear convergence rate sharper than O(1/k).

Exhibits R-linear convergence for generic tensors.

Abstract

Due to the multi-linearity of tensors, most algorithms for tensor optimization problems are designed based on the block coordinate descent method. Such algorithms are widely employed by practitioners for their implementability and effectiveness. However, these algorithms usually suffer from the lack of theoretical guarantee of global convergence and analysis of convergence rate. In this paper, we propose a block coordinate descent type algorithm for the low rank partially orthogonal tensor approximation problem and analyse its convergence behaviour. To achieve this, we carefully investigate the variety of low rank partially orthogonal tensors and its geometric properties related to the parameter space, which enable us to locate KKT points of the concerned optimization problem. With the aid of these geometric properties, we prove without any assumption that: (1) Our algorithm converges globally to a KKT point; (2) For any given tensor, the algorithm exhibits an overall sublinear convergence with an explicit rate which is sharper than the usual $O(1/k)$ for first order methods in nonconvex optimization; {(3)} For a generic tensor, our algorithm converges $R$-linearly.