Approximation and learning with compositional tensor trains

TL;DR

This paper introduces compositional tensor trains (CTTs), a flexible tensor-based model for approximating multivariate functions, combining the expressivity of deep neural networks with efficient tensor algebra for scalable learning.

Contribution

The paper proposes CTTs, a novel compositional tensor train format that unifies various approximation tools and develops new optimization algorithms inspired by natural gradient descent and optimal control.

Findings

CTTs can encode polynomials, DNNs, and tensor networks with similar complexity.

Numerical experiments show CTTs' high expressivity and efficient optimization.

The proposed algorithms outperform traditional methods in regression tasks.

Abstract

We introduce compositional tensor trains (CTTs) for the approximation of multivariate functions, a class of models obtained by composing low-rank functions in the tensor-train format. This format can encode standard approximation tools, such as (sparse) polynomials, deep neural networks (DNNs) with fixed width, or tensor networks with arbitrary permutation of the inputs, or more general affine coordinate transformations, with similar complexities. This format can be viewed as a DNN with width exponential in the input dimension and structured weights matrices. Compared to DNNs, this format enables controlled compression at the layer level using efficient tensor algebra. On the optimization side, we derive a layerwise algorithm inspired by natural gradient descent, allowing to exploit efficient low-rank tensor algebra. This relies on low-rank estimations of Gram matrices, and tensor structured random sketching. Viewing the format as a discrete dynamical system, we also derive an optimization algorithm inspired by numerical methods in optimal control. Numerical experiments on regression tasks demonstrate the expressivity of the new format and the relevance of the proposed optimization algorithms. Overall, CTTs combine the expressivity of compositional models with the algorithmic efficiency of tensor algebra, offering a scalable alternative to standard deep neural networks.