Automatic differentiation for Riemannian optimization on low-rank matrix and tensor-train manifolds

TL;DR

This paper introduces an automatic differentiation-based method to efficiently compute Riemannian gradients and Hessian-vector products for low-rank matrix and tensor-train optimization, simplifying implementation in scientific computing and machine learning.

Contribution

It presents a novel approach leveraging automatic differentiation to compute Riemannian derivatives without requiring explicit formulas, enhancing practical low-rank optimization.

Findings

Efficient computation of Riemannian gradients achieved.

Hessian-vector products can be approximated effectively.

Method simplifies implementation in low-rank optimization tasks.

Abstract

In scientific computing and machine learning applications, matrices and more general multidimensional arrays (tensors) can often be approximated with the help of low-rank decompositions. Since matrices and tensors of fixed rank form smooth Riemannian manifolds, one of the popular tools for finding low-rank approximations is to use Riemannian optimization. Nevertheless, efficient implementation of Riemannian gradients and Hessians, required in Riemannian optimization algorithms, can be a nontrivial task in practice. Moreover, in some cases, analytic formulas are not even available. In this paper, we build upon automatic differentiation and propose a method that, given an implementation of the function to be minimized, efficiently computes Riemannian gradients and matrix-by-vector products between an approximate Riemannian Hessian and a given vector.