Natural Riemannian gradient for learning functional tensor networks

TL;DR

This paper introduces a natural Riemannian gradient descent method for training low-rank functional tensor networks, improving convergence across various loss functions.

Contribution

It extends natural gradient techniques to functional tensor networks, enabling efficient optimization beyond least-squares regression.

Findings

Natural Riemannian gradient improves convergence over standard methods.

Proposed approximations make the approach practical for real datasets.

Numerical experiments demonstrate effectiveness on classification tasks.

Abstract

We consider machine learning tasks with low-rank functional tree tensor networks (TTN) as the learning model. While in the case of least-squares regression, low-rank functional TTNs can be efficiently optimized using alternating optimization, this is not directly possible in other problems, such as multinomial logistic regression. We propose a natural Riemannian gradient descent type approach applicable to arbitrary losses which is based on the natural gradient by Amari. In particular, the search direction obtained by the natural gradient is independent of the choice of basis of the underlying functional tensor product space. Our framework applies to both the factorized and manifold-based approach for representing the functional TTN. For practical application, we propose a hierarchy of efficient approximations to the true natural Riemannian gradient for computing the updates in the parameter space. Numerical experiments confirm our theoretical findings on common classification datasets and show that using natural Riemannian gradient descent for learning considerably improves convergence behavior when compared to standard Riemannian gradient methods.