Optimization on the Extended Tensor-Train Manifold with Shared Factors

TL;DR

This paper develops Riemannian optimization algorithms for tensors in the Extended Tensor Train format with shared factors, addressing the challenges posed by factor constraints and demonstrating practical effectiveness in approximation and eigenvalue problems.

Contribution

It introduces a smooth manifold framework and efficient Riemannian algorithms for ETT tensors with shared factors, a novel approach in tensor optimization.

Findings

Algorithms effectively handle shared factor constraints.

Successful tensor approximation and eigenvalue computations.

Manifold smoothness is theoretically established.

Abstract

This paper studies tensors that admit decomposition in the Extended Tensor Train (ETT) format, with a key focus on the case where some decomposition factors are constrained to be equal. This factor sharing introduces additional challenges, as it breaks the multilinear structure of the decomposition. Nevertheless, we show that Riemannian optimization methods can naturally handle such constraints and prove that the underlying manifold is indeed smooth. We develop efficient algorithms for key Riemannian optimization components, including a retraction operation based on quasi-optimal approximation in the new format, as well as tangent space projection using automatic differentiation. Finally, we demonstrate the practical effectiveness of our approach through tensor approximation tasks and multidimensional eigenvalue problem.

Tables

Table 1. Table 1: Minimum and maximum relative difference in L 2 2 errors among different number of parameters for each number of shared factors between SF-ETT approximation of a 12 12 -tensor of f ​ ( x ) = 1 / ( c 1 ​ x 1 + ⋯ + c 12 ​ x 12 ) f(x)=1/(c_{1}x_{1}+\dotsm+c_{12}x_{12}) , c i = i + 1 c_{i}=i+1 values on a uniform grid of 512 512 points for every argument with and without RStGD

Shared Factors	Min Relative Difference	Max Relative Difference
no sharing	$9.48\times {10}^{-7}$	$9.84\times {10}^{-4}$
3	$3.97\times {10}^{-6}$	$3.59\times {10}^{-3}$
5	$9.47\times {10}^{-6}$	$3.66\times {10}^{-2}$
7	$1.85\times {10}^{-5}$	$2.15\times {10}^{-3}$
9	$1.64\times {10}^{-6}$	$1.23\times {10}^{-2}$
11	$4.81\times {10}^{-5}$	$5.25\times {10}^{-3}$

Table 2. Table 2: Minimum and maximum relative difference in L 2 2 errors among different number of parameters for each number of shared factors between SF-ETT approximation of f ​ ( x ) = exp ⁡ ( − α ​ x 2 ) , α = 0.1 f(x)=\exp(-\alpha x^{2}),\alpha=0.1 values on a uniform grid of 32 10 32^{10} nodes with and without RStGD

Shared Factors	Min Relative Difference	Max Relative Difference
no sharing	$4.28\times {10}^{-13}$	$3.73\times {10}^{-6}$
3	$4.28\times {10}^{-13}$	$3.71\times {10}^{-6}$
5	$4.29\times {10}^{-13}$	$3.83\times {10}^{-6}$
7	$4.27\times {10}^{-13}$	$1.95\times {10}^{-6}$
9	$2.25\times {10}^{-11}$	$9.23\times {10}^{-6}$

Equations

$${\mathcal{X}}_{i_1,\dots ,i_d}=\mathcal{X}(i_1,\dots ,i_d)\in \mathbb{R},i_j\in \overline{1,n_j},j\in \overline{1,d},$$

$$\Vert \mathcal{X}{\Vert }_F^2=\sum _{i_1,\dots ,i_d}^{n_1,\dots ,n_d}{\mathcal{X}}_{i_1,\dots ,i_d}^2,⟨\mathcal{X},\mathcal{Y}⟩=\sum _{i_1,\dots ,i_d}^{n_1,\dots ,n_d}{\mathcal{X}}_{i_1,\dots ,i_d}{\mathcal{Y}}_{i_1,\dots ,i_d}.$$

$${\left({\mathcal{X}}_{(\mu )}\right)}_{i_{\mu },i_{\ne \mu }}={\mathcal{X}}_{i_1,\dots ,i_{\mu },\dots ,i_d},$$

$$i_{\ne \mu }=\sum _{\nu =1,\nu \ne \mu }^d\left(i_{\nu }\prod _{\tau =1,\tau \ne \mu }^{\nu -1}{n}_{\tau }\right).$$

$${\left({\mathcal{X}}^{<\mu >}\right)}_{i_{\text{row}},i_{\text{col}}}={\mathcal{X}}_{i_1,\dots ,i_d},$$

$$i_{\text{row}}=\sum _{\nu =1}^{\mu }\left(i_{\nu }\prod _{\tau =1}^{\nu -1}{n}_{\tau }\right),i_{\text{col}}=\sum _{\nu =\mu +1}^d\left(i_{\nu }\prod _{\tau =\mu +1}^{\nu -1}{n}_{\tau }\right).$$

$$L(W)=W^{<2>}\in {\mathbb{R}}^{n_1{n}_2\times n_3},R(W)=W^{<1>}\in {\mathbb{R}}^{n_1\times n_2{n}_3}.$$

$$L(W{)}^{\top }L(W)=I_{n_3},\text{or}R(W)R(W{)}^{\top }=I_{n_1}.$$

$${\mathcal{X}}_{i_1,\dots ,i_d}=\sum _{j_1,\dots ,j_d}^{r_1^t,\dots ,r_d^t}{G}_{j_1,\dots ,j_d}{U}_{i_1,j_1}^{(1)}\dots U_{i_d,j_d}^{(d)},$$

$$r_i^t=\mathrm{rank}({\mathcal{X}}_{(i)}),i\in \overline{1,d}.$$

$$\mathcal{X}=\left\llbracket\mathcal{G};U^{(1)},\dotsc,U^{(d)}\right\rrbracket.$$

$${\mathcal{X}}_{i_1,\dots ,i_d}=W^{(1)}(i_1)W^{(2)}(i_2)\dots W^{(d)}(i_d),$$

$$W^{(j)}(i_j)\in {\mathbb{R}}^{r_{j-1}^{tt}\times r_j^{tt}},i_j\in \overline{1,n_j},j\in \overline{1,d},r_0^{tt}=r_d^{tt}=1.$$

$$r_i^{tt}=\mathrm{rank}\left({\mathcal{X}}^{<i>}\right),i\in \overline{1,d-1}.$$

$$\mathcal{X}=\mathcal{T}\left(W^{(1)},\dots ,W^{(d)}\right).$$

$${\mathcal{X}}_{\le \mu }$$

$${\mathcal{X}}_{\ge \mu }$$

$${\mathcal{X}}_{\ne \mu }$$

$$\mathcal{X}=\left\llbracket\mathcal{T}\left(W^{(1)},\dotsc,W^{(d)}\right);U^{(1)},\dotsc,U^{(d)}\right\rrbracket.$$

$$r_i^{tt}=\mathrm{rank}\left(A^{<i>}\right),i\in \overline{1,d-1},$$

$$r_j^t=\mathrm{rank}\left(A_{(j)}\right),j\in \overline{1,d}.$$

$$\mathcal{X}=\left\llbracket\mathcal{G};U^{(1)},\dotsc,U^{(d_{t})},U,\dotsc,U\right\rrbracket,$$

$$r_s^t=\mathrm{rank}\left({\mathcal{X}}_{(d_t+1)}|{\mathcal{X}}_{(d_t+2)}|\dots |{\mathcal{X}}_{(d)}\right).$$

$$\mathcal{X}=\left\llbracket\mathcal{T}\left(W^{(1)},\dotsc,W^{(d)}\right);U^{(1)},\dotsc,U^{(d_{t})},U,\dotsc,U\right\rrbracket,$$

$$r_i^{\mathrm{tt}}=\mathrm{rank}\left({\mathcal{X}}^{<i>}\right),\forall i\in \overline{1,d-1},$$

$$r_i^{\mathrm{t}}=\mathrm{rank}\left({\mathcal{X}}_{(i)}\right),\forall i\in \overline{1,d_t},$$

$$r_s^{\mathrm{t}}=\mathrm{rank}\left(\left[\begin{array}{cccc}{\mathcal{X}}_{(d_t+1)}& {\mathcal{X}}_{(d_t+2)}& \dots & {\mathcal{X}}_{(d)}\end{array}\right]\right).$$

$$\mathcal{X}=\left\llbracket\mathcal{G};U^{(1)},\dotsc,U^{(d_{t})},U,\dotsc,U\right\rrbracket,\quad\mathcal{G}=\mathcal{T}\left(W^{(1)},\dotsc,W^{(d)}\right).$$

$${\mathcal{X}}_{(k)}=U^{(k)}{\mathcal{G}}_{(k)}{\left(U{\otimes }_K\cdots {\otimes }_K{U}^{(k+1)}{\otimes }_K{U}^{(k-1)}{\otimes }_K\cdots {\otimes }_K{U}^{(1)}\right)}^{\top },$$

$$\mathrm{rank}\left({\mathcal{X}}_{(k)}\right)\le r_k^t.$$

$$\mathcal{G}=\left\llbracket\mathcal{X};{U^{(1)}}^{\top},\dotsc,{U^{(d_{t})}}^{\top},U^{\top},\dotsc,U^{\top}\right\rrbracket\in\mathbb{R}^{r^{t}_{1}\times\dotsm\times r^{t}_{d_{t}}\times r^{t}_{s}\times\dotsm\times r^{t}_{s}}.$$

$$\begin{gathered} {\mathcal{X}}^{<k>}=\left[\left(U^{(k)}{\otimes }_K\cdots {\otimes }_K{U}^{(1)}\right){\mathcal{G}}_{\le k}\right] \\ \left[{\mathcal{G}}_{\ge k+1}^{\top }\left({\left(U^{(d)}\right)}^{\top }{\otimes }_K\cdots {\otimes }_K{\left(U^{(k+1)}\right)}^{\top }\right)\right], \end{gathered}$$

$$\mathrm{rank}\left({\mathcal{X}}^{<k>}\right)=\mathrm{rank}\left({\mathcal{G}}^{<k>}\right)\le r_k^{tt}.$$

Full text

MSC Classification]53Z50,65F15,15A23,15A69

1 Introduction

Tensor structures play a fundamental role in modern scientific computing and machine learning applications. Among various approaches for handling multidimensional tensors, tensor decompositions have emerged as particularly effective tools. A widely used decomposition is the Tensor Train (TT) format [30], which represents a tensor as a chain of smaller three-dimensional tensors called TT-cores. An important modification, the Extended Tensor Train (ETT) decomposition, further separates physical indices from the TT-cores into distinct factors. From a structural perspective, the ETT representation can be viewed as a special case of the Hierarchical Tucker (HT) decomposition [11], with specific constraints on its factorization pattern. The ETT decomposition proves especially useful when working with tensors exhibiting large mode sizes, offering enhanced computational efficiency.

This work focuses on a constrained version of the decomposition where we enforce equality between selected factors. We refer to this new structure as the Shared-Factor Extended Tensor Train (SF-ETT) decomposition. We demonstrate empirically that factor sharing enables more compact representations and greater computational efficiency in suitable applications. Beyond symmetric tensors, this approach reduces parameter counts when approximating tensorized (quantized) representations [29, 15] and other functions exhibiting similar behavior across modes, outperforming non-shared decompositions.

On the other hand, the factor equality constraint introduces algorithmic challenges. Most significantly, it breaks the multilinearity of the tensor format, which is fundamental to many efficient decomposition algorithms. For instance, the alternating least squares (ALS) approach [12] typically fixes all cores except one, resulting in a linear tensor space. This allows for solving standard low-dimensional least squares problems to update individual cores. However, this property no longer holds in the shared-factor scenario.

Contributions

To address the aforementioned challenges, we leverage another fundamental property of tensor decompositions – their structure as smooth manifolds. This geometric perspective enables the application of Riemannian optimization techniques. In particular, in this work, we specifically prove that the shared-factor structure maintains the smooth manifold property – that is, the set of fixed-rank SF-ETT tensors constitutes a smooth manifold.

We derive efficient algorithms for the essential components of Riemannian optimization in the SF-ETT framework. Specifically, we develop the SF-ETT-SVD algorithm, which constructs approximations in the SF-ETT format through an efficient SVD-based procedure and is applicable for any number of shared factors. We prove that these approximations satisfy quasi-optimality bounds, making them suitable for use as retractions in Riemannian optimization. Another crucial component is the tangent space projection, an essential operation for computing Riemannian gradients and implementing vector transport. We derive efficient projection formulas and demonstrate how to implement it within automatic differentiation frameworks. This approach ensures both generality and broad applicability across various functionals and practical scenarios.

The resulting framework is implemented as an open-source pytorch package that naturally supports GPU parallelization and automatic differentiation. The framework is tested on several examples, including function approximation and multidimensional eigenvalue problem.

Outline

Section 3 introduces the essential tensor notation and formats that are used later for the SF-ETT decomposition. In Section 4, we introduce the SF-ETT decomposition and define the concept of SF-ETT rank. Building on this, Section 4.2 presents Algorithm 1, which constructs a quasi-optimal SF-ETT approximation via matrix truncations.

To enable efficient optimization over SF-ETT representations, we develop a Riemannian framework. Namely, Section 5.1 characterizes the smooth manifold of tensors with fixed SF-ETT rank. Section 5.2 derives the associated tangent space structure, while Section 5.3 provides explicit formulas for the projection operator onto this space. Section 6 consolidates these components into a Riemannian optimization framework, including closed-form expressions for the Riemannian gradient, numerically stable retraction and an automatic differentiation (autodiff)-based implementation for calculation of vector transport and Riemannian gradient.

Finally, in Section 7, we demonstrate the efficacy of our framework through numerical experiments. Section 7.1 presents its application to the approximation of functions on uniform grids, and Section 7.2 showcases its use for solving eigenvalue problems.

2 Related work

The smooth manifold structure is inherent in various tensor decompositions, such as the Tucker decomposition [40], TT [13], and the more general tree-based HT tensor format [42]. However, this property does not hold for the CP decomposition [10] or, more generally, for tensor networks containing loops [8] such as the Tensor Chain (Tensor Ring) format [15, 43]. For more details concerning tensor formats see [16, 14, 7]. For tensor formats that lead to smooth structures one may use the Riemannian optimization techniques [1, 5], which lead to efficient and potentially parallelizable methods.

Tensor-based Riemannian optimization approaches proved useful in different applications. For example, they have been used for matrix and tensor completion [19], linear systems [20] as well as for the multidimensional eigenvalue problems [36]. Low-rank Riemannian optimization has also recently emerged in machine learning applications, see [34] for factorization models and [4] for low-rank adaptation in large language models. The need for Riemannian optimization frameworks that support GPU along with automatic differentiation has led to the development of t3f, ttax, tntorch software packages [23, 24, 41]. Classic python implementation of Riemannian tools is also available in the general-purpose TT package ttpy [31].

The idea of sharing factors was considered in the RESCAL model [26], a relaxed variant of DEDICOM [9], which enforces equal factors in a Tucker-2 decomposition through a heuristic approach. We also mention the work [28], which used shared basis for TT cores and [34], which considered shared factors in the Tucker decomposition context. The authors of [2] seek for the solution of the knowledge graph completion problem with shared factors and the gradient-based approach. In all these works, sharedness has demonstrated significant utility in reducing the total number of parameters while effectively extracting key features from modes with analogous semantic or structural roles.

3 Preliminaries

The primary object of consideration in this work is a concept of tensor as multidimensional array of real numbers. Let $\mathcal{X}\in\mathbb{R}^{n_{1}\times\dotsc\times n_{d}}$ be a $d$-dimensional tensor then by

[TABLE]

we denote its element in the position $i_{1},\dotsc,i_{d}$. The number of elements in a tensor grows exponentially with its dimension $d$, motivating the use of tensor decompositions to achieve substantial memory reduction. The Frobenius tensor norm and the scalar tensor product are defined as follows

[TABLE]

The Kronecker product is denoted as $\otimes_{K}$, the Moore-Penrose pseudo-inverse of matrix $A$ is denoted as $A^{\dagger}$, and the Hadamard product (elementwise) is denoted as $\odot$.

Before presenting specific tensor formats, we first discuss various methods for reorganizing tensor elements into matrix form, as the properties of tensor decompositions fundamentally depend on these matrix representations.

3.1 Tensor matricizations and unfoldings

Let us consider two ways of transforming a tensor into a matrix which shed light on multiple properties of tensor decompositions. The $\mu$-th matricization $\mathcal{X}_{(\mu)}\in\mathbb{R}^{n_{\mu}\times\prod n_{j}/n_{\mu}}$ [39] is a matrix with elements

[TABLE]

where

[TABLE]

This way of mapping a tensor into a matrix is used in the concept of Tucker format defined below. Another way is a construction of the $\mu$-th unfolding of a tensor $\mathcal{X}$ denoted as $\mathcal{X}^{<\mu>}\in\mathbb{R}^{(n_{1}n_{2}\dots n_{\mu})\times(n_{\mu+1}\dots n_{d})}$ [39]. The $\mu$-th unfolding is a matrix with elements

[TABLE]

where

[TABLE]

This transformation is connected to the concept of the TT format that are also described below. Working with tensors in such a format implies working with $3$-dimensional tensors (TT-cores), so it is convenient to use specific notation for them. Let $W\in\mathbb{R}^{n_{1}\times n_{2}\times n_{3}}$ be a $3$-dimensional tensor. Then the unfoldings $W^{<2>}$ and $W^{<1>}$ are called respectively left and right unfoldings of $W$: denoted as

[TABLE]

The tensor $W$ is called left or right orthogonal if respectively

[TABLE]

3.2 Tensor formats

This section is devoted to the definition of tensors format that we ues in the paper and their basic properties. Tensor formats have simple and useful graphical representations that are called tensor diagrams, see Figure 1. A node with $d$ legs represents a $d$-dimensional tensor. Contraction of two tensors along an index is represented by two nodes with a common leg. To also denote orthogonality, we use half-filled nodes: contracting a tensor with itself along nodes from a white-filled region must produce the identity tensor.

3.2.1 Tucker format

The Tucker format [40] of a $d$-dimensional tensor $\mathcal{X}\in\mathbb{R}^{n_{1}\times\dotsc\times n_{d}}$ with rank parameters $(r^{t}_{1},\dotsc,r^{t}_{d})$ is its representation as a contraction of $d$ Tucker factors $U^{(i)}\in\mathbb{R}^{n_{i}\times r^{t}_{i}},i\in\overline{1,d}$ with a Tucker core $\mathcal{G}\in\mathbb{R}^{r^{t}_{1}\times\dotsc\times r^{t}_{d}}$:

[TABLE]

see the corresponding diagram in Figure 2.

There exists a tuple of parameters $(r^{t}_{1},\dotsc,r^{t}_{d})$ with simultaneously attained minimal possible $r^{t}_{i}$. Such tuple is called the Tucker rank of a tensor $\mathcal{X}$ and the minimal $r^{t}_{i}$ can be represented as a rank of the corresponding matricization $\mathcal{X}_{(i)}$ of the tensor $\mathcal{X}$ defined in (1)

[TABLE]

The Tucker decomposition (3) is further formally denoted as

[TABLE]

Note that the dimension of the Tucker core equals to the dimension of the tensor. As a result, this format suffers from the exponential growth of the number of parameters – manifestation of the the curse of dimensionality.

3.2.2 Tensor Train format

The TT decomposition [30, 32] of a tensor $\mathcal{X}\in\mathbb{R}^{n_{1}\times\dotsc\times n_{d}}$ with the rank parameters $(r^{tt}_{1},\dotsc,r^{tt}_{d-1})$ is defined as:

[TABLE]

where

[TABLE]

The corresponding tensor diagram is presented in Figure 3.

There exists a tuple of rank parameters $(r^{tt}_{1},\dotsc,r^{tt}_{d-1})$ with simultaneously attainable minimal possible $r^{tt}_{i}$. Such a tuple is called the TT-rank of a tensor $\mathcal{X}$. The minimal possible values $r^{tt}_{i}$ can be expressed as a rank of corresponding unfolding $\mathcal{X}^{<i>}$ of tensor $\mathcal{X}$ defined in (2):

[TABLE]

The TT decomposition will be further denoted as

[TABLE]

For the TT decomposition, one can define auxiliary useful tensors [22]:

[TABLE]

In contrast to the Tucker decomposition, when the ranks are bounded, we observe that the TT format has polynomial growth of the total number of parameters, breaking the curse of dimensionality.

3.2.3 Extended Tensor Train format

The ETT decomposition format of a tensor $\mathcal{X}\in\mathbb{R}^{n_{1}\times\dotsc\times n_{d}}$ with the rank parameters $(r^{tt}_{1},\dotsc,r^{tt}_{d-1},r^{t}_{1}\dotsc,r^{t}_{d})$ can be viewed as a combination the Tucker and the ETT formats. In particular, it is a contraction of a Tucker core represented in the TT format with $d$ Tucker factors $U^{(i)}\in\mathbb{R}^{n_{i}\times r_{i}^{t}}$:

[TABLE]

One can also notice that The ETT format is a specific case of the HT decomposition [42]. The tensor diagram for (9) is presented in Figure 4.

Similarly, the rank values $(r^{tt}_{1},\dotsc,r^{tt}_{d-1},r^{t}_{1}\dotsc,r^{t}_{d})$ with the minimal possible values are:

[TABLE]

3.2.4 Shared Factor Tucker decomposition

The SF-Tucker decomposition [34] of a $d$-dimensional ($d=d_{t}+d_{s}$) tensor $\mathcal{X}\in\mathbb{R}^{n_{1}\times\dotsc\times n_{d_{t}}\times n_{s}\times\dotsc\times n_{s}}$ with the rank parameters $(r^{t}_{1},\dotsc,r^{t}_{d_{t}},r^{t}_{s})$ is its representation as a contraction of $d_{t}$ Tucker factors $U^{(i)}\in\mathbb{R}^{n_{i}\times r^{t}_{i}},i\in\overline{1,d_{t}}$, one shared Tucker factor $U\in\mathbb{R}^{n_{s}\times r^{t}_{s}}$ (repeated $d_{s}$ times) and the Tucker core $\mathcal{G}\in\mathbb{R}^{r^{t}_{1}\times\dotsc\times r^{t}_{d_{t}}\times r^{t}_{s}\times\dotsc\times r^{t}_{s}}$:

[TABLE]

the corresponding tensor diagram is in Figure 5.

The rank is defined similarly to the Tucker case, but with the additional constraint of equality for some of the modes. As a result, we have a tuple of $d_{t}$ rank parameters and the additional shared one: $(r^{t}_{1},\dotsc,r^{t}_{d_{t}},r^{t}_{s})$ with minimal $r^{t}_{i}$. The ranks $r^{t}_{i},i\in\overline{1,d_{t}}$ coincide with the respective Tucker ranks, while the shared rank $r^{t}_{s}$ can be found as follows:

[TABLE]

4 Shared-Factor Extended Tensor Train decomposition

In this section, we present the new SF-ETT decomposition. For a $d$-dimensional tensor $\mathcal{X}\in\mathbb{R}^{n_{1}\times\dotsc\times n_{d_{t}}\times n_{s}\times\dotsc\times n_{s}}$, its SF-ETT decomposition reads as:

[TABLE]

where $W^{(i)}\in\mathbb{R}^{r^{\mathrm{tt}}_{i-1}\times r^{\mathrm{t}}_{i}\times r^{\mathrm{tt}}_{i}}$, $i\in\overline{1,d_{t}}$, $W^{(i)}\in\mathbb{R}^{r^{\mathrm{tt}}_{i-1}\times r^{\mathrm{t}}_{s}\times r^{\mathrm{tt}}_{i}}$, $i\in\overline{d_{t}+1,d}$ ($r^{\mathrm{tt}}_{0}=r^{\mathrm{tt}}_{d}=1$), $U^{(j)}\in\mathbb{R}^{n_{j}\times r^{\mathrm{t}}_{j}},j\in\overline{1,d_{t}}$, and $U\in\mathbb{R}^{n_{s}\times r^{\mathrm{t}}_{s}}$ – the shared factor. The tensor diagram for (12) is presented in Figure 6.

4.1 SF-ETT ranks

Now we need to define the rank for this format. By analogy to the other tensor formats, we have that all rank parameters can be minimized independently. Formal verification of this fact is given in the proof the Theorem 1. Thus, the tuple $(r^{\mathrm{tt}}_{1},\dotsc,r^{\mathrm{tt}}_{d-1},r^{\mathrm{t}}_{1},\dots r^{\mathrm{t}}_{d_{t}},r^{\mathrm{t}}_{s})$ is called the SF-ETT rank if it consists of minimal possible values. The exact values in terms of unfolding matrices is given in the theorem below.

Theorem 1**.**

The SF-ETT rank $(r^{\mathrm{tt}}_{1},\dotsc,r^{\mathrm{tt}}_{d-1},r^{\mathrm{t}}_{1},\dots r^{\mathrm{t}}_{d_{t}},r^{\mathrm{t}}_{s})$ of a tensor $\mathcal{X}$ is given by

[TABLE]

Proof.

Indeed, let $\mathcal{X}$ tensor be represented in the SF-ETT format

[TABLE]

Due to the fact that SF-ETT format is a specific case of the Tucker decomposition, the following equation holds

[TABLE]

which is a skeleton decomposition. This means that

[TABLE]

The shared Tucker rank formula arises from the same proposition in [34]. As a result, one may take $U^{(i)}\in\mathbb{R}^{n_{i}\times r^{t}_{i}}$ as basis columns in $\mathrm{Im}\,\left(\mathcal{X}_{(i)}\right)$ for every $i\in\overline{1,d_{t}}$ and $U\in\mathbb{R}^{n\times r^{t}_{s}}$ as basis columns in $\mathrm{Im}\left(\begin{bmatrix}\mathcal{X}_{(d_{t}+1)}&\mathcal{X}_{(d_{t}+2)}&\dots&\mathcal{X}_{(d)}\end{bmatrix}\right)$ then the Tucker core can be calculated as follows:

[TABLE]

On the other hand, the formula for the $k$-th unfolding in the ETT format is

[TABLE]

where $U^{(k)}=U,k>d_{t}$ and explicit formulas for $\mathcal{G}_{\leq\mu},\mathcal{G}_{\geq\mu}$ written in (7), (8). It is a skeleton decomposition for the unfolding. Moreover from (13), due to the full column rank of $U^{(i)}$ one can notice that

[TABLE]

Decomposing the core $\mathcal{G}$ into the TT decomposition with the TT rank equal to $(\mathcal{X}^{<1>},\dotsc,\mathcal{X}^{<d-1>})$ [33], one may obtain the SF-ETT decomposition with the parameters that achieve the proposed lower bounds. ∎

The SF-ETT decomposition is not unique and can be stored in orthogonalized form. A decomposition of $\mathcal{X}$ that consists of orthogonal Tucker factors $U,U^{(i)},i\in\overline{1,d_{t}}$ and $\mu$-orthogonal TT decomposition of Tucker core is denoted as a $\mu$-orthogonal SF-ETT form

[TABLE]

where $W_{L}^{(i)},i\in\overline{1,\mu-1}$ are left orthogonal, $W_{R}^{(i)},i\in\overline{\mu+1,d}$ are right orthogonal and $\overline{W}^{(\mu)}$ is unconstrained. This can be achieved via applying successive QR decomposition operations to Tucker factors and TT cores. The tensor diagram for the $\mu$-orthogonal SF-ETT tensor form is represented in (7).

Here and further all SF-ETT decomposed tensors should be considered as $(d-1)$-orthogonal unless otherwise stated. The amount of parameters to store SF-ETT decomposed tensor $\mathcal{X}$ is

[TABLE]

Suppose that $d_{s}=d$, i.e., all the modes are shared. For simplicity let us also assume that all the ranks are equal to $r$. We are interested in the regime when the mode size $n_{s}$ is large, meaning that $n_{s}\gg r^{2}$. From (15), we observe that the benefit in storage for the proposed decomposition is proportional to the dimensionality $d$.

4.2 SF-ETT-SVD and rounding

Next, we need a reliable algorithm to approximate a tensor in the new format. Fortunately, such an algorithm exists and boils down to a number of SVD decompositions. We refer to this algorithm as the SF-ETT-SVD. This algorithm is also quasi-optimal as follows from Theorem 2, meaning that it can be further used as a retraction in the Riemannian optimization framework.

Theorem 2**.**

Let $\mathcal{X}\in\mathbb{R}^{n_{1}\times\dotsc\times n_{d_{t}}\times n_{s}\times\dotsc\times n_{s}}$, let also $U^{(i)}\in\mathbb{R}^{n_{i}\times r^{t}_{i}}$ be a matrix of the first $r^{t}_{i}$ left singular vectors of $\mathcal{X}_{(i)}$ and $U\in\mathbb{R}^{n_{s}\times r^{t}_{s}}$ of the first $r^{t}_{s}$ left singular vectors of $\begin{bmatrix}\mathcal{X}_{(d_{t}+1)}&\mathcal{X}_{(d_{t}+2)}&\dots&\mathcal{X}_{(d)}\end{bmatrix}$. Let $\mathcal{G}\in\mathbb{R}^{r^{t}_{1}\times\dotsm r^{t}_{s}}$ such that

[TABLE]

Let also $\mathcal{T}\left(W^{(1)},\dotsc,W^{(d)}\right)$ be the TT-SVD [30] approximation of $\mathcal{G}$ with the TT-rank $(r^{tt}_{1},\dots,r^{tt}_{d-1})$. Then for $\mathbf{r}=(r^{\mathrm{tt}}_{1},\dotsc,r^{\mathrm{tt}}_{d-1},r^{\mathrm{t}}_{1},\dots r^{\mathrm{t}}_{d_{t}},r^{\mathrm{t}}_{s})$ the tensor

[TABLE]

is a quasi-optimal approximation of $\mathcal{X}$:

[TABLE]

where $\preceq$ is the element-wise comparison and $C(d)=\left(\sqrt{d}+\sqrt{d}\sqrt{d-1}+\sqrt{d-1}\right).$

Proof.

Let $\mathcal{X}\in\mathbb{R}^{n_{1}\times\dotsc\times n_{d_{t}},n_{s},\dotsc,n_{s}}$ be an arbitrary tensor and $P_{\mathbf{r}}^{\mathtt{sf-ett-hosvd}}(\mathcal{X})$ denoted as $P(\mathcal{X})$ its SF-ETT approximation of rank $\mathbf{r}=(\mathbf{r}^{\mathrm{tt}},\mathbf{r}^{\mathrm{t}})$ implemented via applying SF-HOSVD to TT compressed tensor denoted as $P_{\mathrm{TT}}(\mathcal{X})$, then

[TABLE]

[TABLE]

Therefore

[TABLE]

Now let $P(\mathcal{X})=P_{\mathbf{r}}^{\mathtt{sf-ett-hosvd}}(\mathcal{X})$ be a SF-ETT approximation of rank $\mathbf{r}=(\mathbf{r}^{\mathrm{tt}},\mathbf{r}^{\mathrm{t}})$ obtained by applying TT compression onto core $\mathcal{G}$ of SF-Tucker approximation $P_{\mathrm{SF}}(\mathcal{X})$ of $\mathcal{X}$ with rank equal to $\mathbf{r}^{\mathrm{t}}$, then

[TABLE]

Now if $P_{\mathrm{SF}}\left(\mathcal{X}\right)=\llbracket\mathcal{G};U^{(1)},\dotsc,U\rrbracket$, where $U^{(1)},\dotsc,U$ — orthogonal matrices then

[TABLE]

Therefore

[TABLE]

∎

Remark 1**.**

The compression algorithm justified by Theorem 2 can be implemented in two ways. The first way is to compress tensor $\mathcal{X}$ into TT decomposition

[TABLE]

and then apply SF-HOSVD algorithm to the tensor

[TABLE]

The second way is to compress the input tensor via SF-HOVSD first and then apply TT compression to the core of the obtained approximation. Both approaches will ensure quasi-optimal approximation with the same quasi-optimality constant.

If one has a tensor already in the SF-ETT format, it is possible to efficiently apply SF-ETT-SVD without forming full tensors. This operation is called SF-ETT-Rounding and has asymptotic complexity

[TABLE]

where $r^{\mathrm{t}}=\max_{i}r^{\mathrm{t}}_{i}$, $r^{\mathrm{tt}}=\max_{j}r^{\mathrm{tt}}_{j}$, $N=\max_{i}n_{i}$.

The SF-ETT Rounding algorithm is described in Algorithm 1 and can be logically split into two stages: TT and SF-Tucker roundings. The first stage is fully described in the first line: TT rank is rounded to the target rank via $\mathtt{TTRound}$ from [33]. All of the other steps are devoted to the second stage.

This stage begins with the computation of unconstrained core $\overline{W}^{(k)}$ in $k$-or-thogonal representation of SF-ETT tensor for every $k\in\overline{1,d}$ defined in (14) (rows $2-6$). According to [34], the algorithm should compute SVD of $\mathcal{X}_{(k)}$ if $k\leq d_{t}$ and SVD of $\begin{bmatrix}\mathcal{X}_{(d_{t}+1)}&\mathcal{X}_{(d_{t}+2)}&\dots&\mathcal{X}_{(d)}\end{bmatrix}$. The main idea is to obtain this decomposition without forming dense Tucker core out of its TT format. Suppose that the tensor $\mathcal{X}$ is stored in the $k$-orthogonal format defined in (14)

[TABLE]

One may write its $k$-th matricization for $k\leq d_{t}$ as follows

[TABLE]

where

[TABLE]

is a matrix with orthonormal rows, so

[TABLE]

This structure is also preserved for the shared Tucker rank:

[TABLE]

Due to orthogonality of $U$ and a block-diagonal matrix on the right hand side, the SVD of $\begin{bmatrix}\mathcal{X}_{(d_{t}+1)}&\mathcal{X}_{(d_{t}+2)}&\dots&\mathcal{X}_{(d)}\end{bmatrix}$ can be obtained from the SVD of $\begin{bmatrix}\overline{W}^{(d_{t}+1)}_{(2)}&\overline{W}^{(d_{t}+2)}_{(2)}&\dots&\overline{W}^{(d)}_{(2)}\end{bmatrix}$. The lines from $7$ to $12$ are devoted to the rounding of SF-Tucker modes corresponding to the non-shared Tucker factors via the SVD truncation. The rounding for the shared SF-Tucker mode is described in the lines $13$ to $18$.

5 Shared Factor Extended Tensor Train manifold

The set of fixed rank $\mathbf{r}^{t}$ Tucker tensors $\mathrm{T}\,(\mathbf{r}^{t})\subseteq\mathbb{R}^{n_{1}\times\dotsc\times n_{d_{t}}\times n_{s}\times\dotsc\times n_{s}}$ forms a smooth embedded submanifold [39]. In what follows, we assume that $\mathbf{r}^{t}=(r^{t}_{1},\dots,r^{t}_{d_{t}},r^{t}_{s},\dots,r^{t}_{s})$ and introduce the notation $\mathbf{r}^{ts}=(r^{t}_{1},\dots,r^{t}_{d_{t}},r^{t}_{s})$ for the SF-Tucker rank. Then the set of fixed rank $\mathbf{r}^{ts}$ SF-Tucker tensors $\mathrm{SFT}\,(\mathbf{r}^{ts})\subset\mathrm{T}\,(\mathbf{r}^{t})$ is also a smooth manifold [34] embedded into the linear space $\mathbb{R}^{n_{1}\times\dotsc\times n_{d_{t}}\times n_{s}\times\dotsc\times n_{s}}$. Analogically $\mathrm{ETT}\,(\mathbf{r}^{tt},\mathbf{r}^{t})\subset\mathrm{T}\,(\mathbf{r}^{t})$ denotes a smooth tensor manifold of fixed ETT rank $(\mathbf{r}^{tt},\mathbf{r}^{t})$, which is the HT manifold [42] embedded into the linear space $\mathbb{R}^{n_{1}\times\dotsc\times n_{d_{t}}\times n_{s}\times\dotsc\times n_{s}}$. Finally, $\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})\subset\mathrm{T}\,(\mathbf{r}^{t})$ stands for the set of tensors of fixed SF-ETT rank $(\mathbf{r}^{tt},\mathbf{r}^{ts})$.

5.1 Smoothness of the SF-ETT manifold

The purpose of this section is to introduce a smooth structure on the set of all tensors that can be represented in the SF-ETT format with the fixed SF-ETT rank.

Lemma 3**.**

Under the rank constraints above, the $\mathrm{SFETT}$ set represents the following intersection:

[TABLE]

Proof.

Let $\mathcal{X}\in\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})$. Then by the definition of the ETT format:

[TABLE]

One can also denote

[TABLE]

so

[TABLE]

This means that:

[TABLE]

Now let $\mathcal{X}\in\mathrm{SFT}\,(\mathbf{r}^{ts})$ and $\mathcal{X}\in\mathrm{ETT}\,(\mathbf{r}^{tt},\mathbf{r}^{t})$:

[TABLE]

where $U^{(1)},\dotsc,U$ have orthonormal columns. Then by the definition of ETT

[TABLE]

Contracting the factors ${U^{(i)}}^{\top}$ and $U^{\top}$ with $W^{(i)}$, one can get

[TABLE]

and, hence, $\mathcal{X}\in\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})$. ∎

The intersection of two smooth manifolds is not necessarily a smooth manifold. Fortunately there is a special type of manifold intersection that preserves smoothness.

Definition 1** (Transversality [21]).**

Let $\mathcal{M}$ be a smooth manifold and $\mathcal{A}$, $\mathcal{B}$ are two embedded submanifolds of $\mathcal{M}$. The $\mathcal{A}\cap\mathcal{B}$ is a transversal intersection if $\;\forall p\in\mathcal{A}\cap\mathcal{B}$:

[TABLE]

Lemma 4** (Transversality of the intersection).**

$\mathrm{SFT}\,(\mathbf{r}^{ts})$* and $\mathrm{ETT}\,(\mathbf{r}^{tt},\mathbf{r}^{t})$ intersect transversally in $\mathrm{T}\,(\mathbf{r}^{t})$, i.e., $\;\forall\mathcal{X}\in\mathrm{SFT}\,(\mathbf{r}^{ts})\cap\mathrm{ETT}\,(\mathbf{r}^{tt},\mathbf{r}^{t})$, we have*

[TABLE]

Proof.

Let $X\in\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})$, then

[TABLE]

According to [42], we have

[TABLE]

i.e., every $\xi\in T_{\mathcal{X}}\mathrm{T}$ can be described via the tuple $\left(\dot{\mathcal{G}},\dot{Z}^{(1)},\dotsc,\dot{Z}^{(d)}\right)$. The tangent space for $\mathrm{SFT}\,(\mathbf{r}^{ts})$ from [34] can be written as

[TABLE]

and, hence, every $\eta\in T_{\mathcal{X}}\mathrm{SFT}$ can be described via the tuple $(\dot{\mathcal{G}},\dot{U}^{(1)},\dotsc,\dot{U}^{(d_{t})},\dot{U})$. The tangent space for $\mathrm{ETT}(\mathbf{r}^{tt},\mathbf{r}^{t})$ is a particular case of the tangent space for manifold of fixed HT rank described in [42]

[TABLE]

so every $\zeta\in T_{\mathcal{X}}\mathrm{ETT}$ can be described via the tuple $\left(\dot{W}^{(1)},\dotsc,\dot{W}^{(d)},\dot{Y}^{(1)},\dotsc,\dot{Y}^{(d)}\right)$.

Now let $\eta\in T_{\mathcal{X}}\mathrm{SFT}$, $\zeta\in T_{\mathcal{X}}\mathrm{ETT}$, then

[TABLE]

and can be interpreted as element of $T_{\mathcal{X}}\mathrm{T}$ in terms of the following tuple

[TABLE]

As a result, we obtain $T_{\mathcal{X}}\mathrm{ETT}+T_{\mathcal{X}}\mathrm{SFT}\subseteq T_{\mathcal{X}}\mathrm{T}$.

Let $\xi$ be an arbitrary element of $T_{\mathcal{X}}\mathrm{T}$ given by the tuple $\left(\dot{\mathcal{G}},\dot{Z}^{(1)},\dotsc,\dot{Z}^{(d)}\right)$, then $\exists\;\hat{\eta}\in T_{\mathcal{X}}\mathrm{SFT},\hat{\zeta}\in T_{\mathcal{X}}\mathrm{ETT}:\hat{\eta}+\hat{\zeta}=\xi$. For example, $\hat{\eta}$ may be given by a tuple

[TABLE]

and $\hat{\zeta}$ by

[TABLE]

So $T_{\mathcal{X}}\mathrm{ETT}+T_{\mathcal{X}}\mathrm{SFT}\supseteq T_{\mathcal{X}}\mathrm{T}$. ∎

Uniting the results of Lemma 3 and Lemma 4, one can obtain the smooth structure on $\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})$.

Theorem 5**.**

The set $\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})$ forms a smooth manifold embedded into $\mathbb{R}^{n_{1}\times\dotsc\times n_{d_{t}}\times n_{s}\times\dotsc\times n_{s}}$ of the dimension

[TABLE]

where $d_{s}=d-d_{t}>0$, $r_{0}^{tt}=r_{d}^{tt}=1$ and

[TABLE]

Proof.

First of all, according to [38, Definition $1.37$], we have that $\mathrm{ETT}\,(\mathbf{r}^{tt},\mathbf{r}^{t})$ and $\mathrm{SFT}\,(\mathbf{r}^{ts})$ are smooth manifolds embedded into the $\mathrm{T}\,(\mathbf{r}^{t})$, which in turn is embedded into $\mathbb{R}^{n_{1}\times\dotsc\times n_{d_{t}}\times n_{s}\times\dotsc\times n_{s}}$. Then we use the fact that $\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})$ is the transversal intersection of these manifolds in $\mathrm{T}(\mathbf{r}^{t})$. Hence, according to [35] $\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})$ is a smooth manifold embedded in $\mathrm{T}(\mathbf{r}^{t})$ with the dimension

[TABLE]

where

[TABLE]

∎

5.2 Tangent space

A tangent space provides a local linear approximation of a smooth manifold at a given point. It consists of tangent vectors, which can be characterized as the first derivatives of smooth curves on the manifold passing through that point:

[TABLE]

The tangent space for the manifold of fixed SF-ETT rank at

[TABLE]

can be written as

[TABLE]

where

[TABLE]

$\dot{W}^{(i)}\in\mathbb{R}^{r^{\mathrm{tt}}_{i-1}\times r^{\mathrm{t}}_{i}\times r^{\mathrm{tt}}_{i}}$, $i\in\overline{1,d_{t}}$, $\dot{W}^{(i)}\in\mathbb{R}^{r^{\mathrm{tt}}_{i-1}\times r^{\mathrm{t}}_{s}\times r^{\mathrm{tt}}_{i}}$, $i\in\overline{d_{t}+1,d}$ ($r^{\mathrm{tt}}_{0}=r^{\mathrm{tt}}_{d}=1$), $\dot{U}^{(j)}\in\mathbb{R}^{n_{j}\times r^{\mathrm{t}}_{j}},j\in\overline{1,d_{t}}$, $\dot{U}\in\mathbb{R}^{n_{s}\times r^{\mathrm{t}}_{s}}$. Let $\mathcal{X}$ also be in the $(d-1)$ orthogonalized form. Then we impose additional gauge conditions:

[TABLE]

It is also possible to store every tangent vector in SF-ETT form with at most doubled SF-ETT rank:

[TABLE]

where $\mathcal{T}\left(\hat{W}^{(1)},\dotsc,\hat{W}^{(d)}\right)\in\mathbb{R}^{2r_{1}\times\dotsc\times 2r_{d_{t}}\times 2r_{s}\times\dotsc\times 2r_{s}}$ and

[TABLE]

Where $\overline{W}^{(i)},i\in\overline{1,d}$ is the $i$-th core in the $i$-th orthogonalization, where $\overline{W}^{(i)}$ is the only core without the orthogonality constraint.

5.3 Projection to the tangent space

The orthogonal projection to a tangent space of a smooth manifold is a crucial step in gradient-based optimization algorithms on smooth manifolds. This section proposes formulas for computing the orthogonal projection onto the tangent space of $\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})$.

Every orthogonal projection is associated with a scalar product, defined on tangent space, called Riemannian metric. Further statements are based on Riemannian metric coinciding with a standard scalar product in $\mathbb{R}^{n_{1}\times\dotsc\times n_{d_{t}}\times n_{s}\times\dotsc\times n_{s}}$ on $T_{\mathcal{X}}\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})$. In this case, the computation of Riemannian gradient can be simplified to the computation of a projection of Euclidean gradient onto the tangent space:

[TABLE]

where $f\colon\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})\to\mathbb{R}$ is a restriction of some smooth function $\overline{f}\colon\mathbb{R}^{n_{1}\times\dots n_{d_{t}}\times n_{s}\times\dots\times n_{s}}\to\mathbb{R}$ to $\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})$. Let $Z\in\mathbb{R}^{n_{1}\times\dotsc\times n_{d_{t}}\times n_{s}\times\dotsc\times n_{s}}$ and $\mathcal{X}\in\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})$:

[TABLE]

then $P_{T_{\mathcal{X}}\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})}\left(Z\right)\in T_{\mathcal{X}}\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})$ (or $P(Z)$ for short) admits the following SF-ETT decomposition:

[TABLE]

where

[TABLE]

Proof.

The justification of the formulas above consists of two steps fully described in Appendix A: the first stage is to notice that tangent space can be decomposed into the sum of mutually orthogonal linear subspaces, then deriving the projection stands behind deriving several projections onto these linear subspaces. ∎

5.4 Retraction

Retraction

[TABLE]

for a smooth manifold $\mathcal{M}$ is a smooth mapping which satisfies the following two conditions [1]:

[TABLE]

The retraction operation approximates exponential maps that are often difficult to compute from the computational viewpoint. The following theorem proposes the SF-ETT-Rounding Algorithm 1 as a retraction for $\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})$.

Proposition 6**.**

The mapping

[TABLE]

where $P_{(\mathbf{r}^{tt},\mathbf{r}^{ts})}^{\mathtt{sf-ett-hosvd}}$ is from Theorem 2 is a retraction for the manifold $\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})$.

Proof.

Using quasi-optimality of Algorithm 1 and [19, Proposition $2.3.$], we immediately obtain the desired result. ∎

6 Gradient-based optimization on the manifold

This section is devoted to the description of the key ingredients of the gradient-based Riemannian optimization algorithm on the manifold of the fixed SF-ETT rank. The details of their application in practice will be given further in Section 7.

Let a scalar-valued function $f\colon\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})\to\mathbb{R}$ be a restriction of some smooth function $\overline{f}\colon~\mathbb{R}^{n_{1}\times\dots\times n_{d_{t}}~\times n_{s}\times\dots\times n_{s}}\to\mathbb{R}$ to $\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})$. We aim to solve the task:

[TABLE]

A loop of the Riemannian gradient descent algorithm for solving (22) can be organized according to the following scheme [1]. First, one computes an optimization direction $h_{k}$ parallel to the Riemannian anti-gradient $(-1)\cdot\mathrm{grad}f(\mathcal{X}_{k})$ at the current point $\mathcal{X}_{k}$. Afterwards the direction is stretched by a step size value $\alpha_{k}>0$. This optimization loop finalizes with a step from $\mathcal{X}_{k}$ towards $\alpha_{k}h_{k}$ along $T_{\mathcal{X}_{k}}\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})$. The result of this operation has to belong to $\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})$. Therefore one may use a retraction operation to satisfy this condition:

[TABLE]

A schematic visualization of the presented optimization loop is shown in Figure 8.

6.1 Riemannian gradient computation

Let

[TABLE]

be in $(d-1)$-orthogonal SF-ETT form. According to (18), the Riemannian gradient computation can be reduced to calculation of a projection of the Euclidean gradient $\nabla\overline{f}(\mathcal{X}_{k})$ to the tangent space:

[TABLE]

Since $\mathrm{grad}f\left(\mathcal{X}_{k}\right)\in T_{\mathcal{X}_{k}}\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})$, one can describe it with tensors

[TABLE]

that form a SF-ETT represented tensor with ranks which do not exceed $(2\mathbf{r}^{tt},2\mathbf{r}^{ts})$ (see Section 5.2).

Mentioning the fact that the Euclidean gradient may have a significantly bigger SF-ETT rank than $(2\mathbf{r}^{tt},2\mathbf{r}^{ts})$ as described in [25] one can come up with the idea of computing the whole projection in (23) at once. That can be accomplished via the automatic differentiation (autodiff) approach. This idea was firstly described in [25]. Let us define a construction operator, which composes the tangent vector as an element of $\mathbb{R}^{n_{1}\times\dots n_{d_{t}}\times n_{s}\times\dots\times n_{s}}$ from its cores and factors:

[TABLE]

One can easily verify that

[TABLE]

and defining a new real-valued function $g$:

[TABLE]

one can compute partial derivatives of $g$ with respect to parameters $S^{(1)},\dotsc,S^{(d)},$ $B^{(1)},\dotsc,B^{(d_{t})},A$ via autodiff:

[TABLE]

where

[TABLE]

Proof.

The justification of these formulas is presented in Appendix B. ∎

This leads to closed-form formulas for computing Riemannian gradient parameters on the tangent space:

[TABLE]

Remark 2**.**

In practice, the implementation of the construction operator from (24) does not perform explicit arithmetic operations. Instead, since the image of $C_{\mathcal{X}}$ belongs to the tangent space $T_{\mathcal{X}}\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})$, the tensor $\mathcal{X}=C_{\mathcal{X}}(\theta)$ is stored in $(2\mathbf{r}^{tt},2\mathbf{r}^{ts})$ SF-ETT format as detailed in Section 5.2. Henceforth, $C_{\mathcal{X}}(\theta)$ is passed into the $\overline{f}$ as standard tensor in SF-ETT format.

6.2 Retraction

As described in Section 6, once the search direction $h_{k}$ is given, the final step of a Riemannian optimization algorithm is a retraction step:

[TABLE]

Proposition 6 states that Algorithm 1 is a retraction for $\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})$:

[TABLE]

Since both tangent foot $\mathcal{X}_{k}$ and minimization direction $\alpha_{k}h_{k}$ belong to the same tangent space, its sum also belongs that linear space, so $(\mathcal{X}_{k}+\alpha_{k}h_{k})$ is a SF-ETT decomposed tensor with at most $(2\mathbf{r}^{tt},2\mathbf{r}^{ts})$ SF-ETT rank that allows for straightforward application of Algorithm 1 without any explicit calculations of dense tensors out of SF-ETT factors.

6.3 Momentum and vector transport

One can enhance the Riemannian gradient descent algorithm by additionally incorporating momentum vectors into the minimization direction. Specifically, the optimization direction $h_{k}$ can be seen as a linear combination of the current Riemannian gradient $\mathrm{grad}f(\mathcal{X}_{k})$ and the accumulation of Riemannian gradients from previous iterations $M_{k}$. Generally every new optimization point has a new tangent space built around it, so each momentum vector $M_{k-1}\in\mathrm{T}_{\mathcal{X}_{k-1}}\mathrm{SFETT}(\mathbf{r}^{tt},\mathbf{r}^{ts})$ needs to be transported to the tangent space at the current point:

[TABLE]

where $\tau_{k}\colon T_{\mathcal{X}_{k-1}}\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})\to T_{\mathcal{X}_{k}}\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})$ is called a vector transport function [1]. In particular, we use a projection onto the current tangent space as a vector transport function:

[TABLE]

The computation of such operation can be reduced to one autodiff call as proposed by differentiating

[TABLE]

since

[TABLE]

Therefore, to compute a projection of any element $\xi$ to a tangent space, one needs to compute a Riemannian gradient of the function $\hat{f}$.

7 Numerical experiments

The developed Riemannian framework is highly versatile, enabling its use across a wide range of functionals and practical problems where tensor decompositions are applicable. In this section, we apply it to two examples: the compression of grid functions and the solution of multidimensional eigenvalue problems, and study the effect of using multiple shared factors.

7.1 Grid functions

This section presents an evaluation of the performance of Algorithm 1 in the context of approximating grid-based functions. The experimental procedure is structured as follows. Initially, a target function is approximated by a tensor $A$ using the TT-cross approximation algorithm applied on a uniform grid $G=\{x_{k}|x_{k}=k/N,N\in\mathbb{R}\}$ [33] with high precision (e.g. $10^{-12}$ for float32 precision). Subsequently, a fixed number of shared Tucker factors are initialized alongside regular Tucker factors, both set as identity matrices, and incorporated into the TT-cross approximated tensor. This yields an exact SF-ETT representation $\hat{A}$ of the tensor $A$, although the associated SF-ETT ranks may not be optimal. To obtain a quasi-optimal SF-ETT approximation $\mathcal{X}_{0}$, Algorithm 1 is applied with the prescribed SF-ETT ranks $(\mathbf{r}^{tt},\mathbf{r}^{ts})$, i.e

[TABLE]

For the optimal approximation, $\mathcal{X}_{0}$ serves as the initial guess for a Riemannian steepest gradient descent (RStGD) method on the manifold $\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})$ (see Section 6) aimed at minimizing the L$2$ error:

[TABLE]

The Riemannian gradient of this objective is given by

[TABLE]

The steepest optimization step $\alpha_{k}^{\star}$ in $\mathcal{X}_{k+1}=\mathcal{X}_{k}-\alpha_{k}\,\mathrm{grad}f(\mathcal{X}_{k})$ is

[TABLE]

The RStGD procedure is stopped unless the next optimization point gives a smaller objective function value.

We apply the proposed approach to approximate $12$-dimensional function defined as:

[TABLE]

on $12$ independent uniform grids $G_{i}$ each consisting of $N_{i}=512$ points:

[TABLE]

Hence, the tensor $\mathcal{X}_{0}$ is a $12$-dimension tensor with mode sizes $n_{i}=512$ for $i\in\overline{1,12}$. For varying numbers of shared factors, we construct $\mathcal{X}_{0}$ with different rank configurations and record the number of parameters in each tensor. Subsequently, we minimize the L$2$ approximation error and depict the relationship between the number of parameters and the error in Figure 9. The steps-like morphology of the curves is a result of the iterative procedure, wherein the Tucker rank and the Tensor Train (TT) rank are increased alternately by one. Table 1 demonstrates relative differences between relative L$2$ error without using RStGD and with it.

The second function considered for approximation is $f(x)=\exp(-\alpha x^{2}),x\in[0,1]$ sampled on a uniform grid with step size $1/32^{10}$. Accordingly, the tensor $A$ is constructed in the QTT format [29, 15] with mode sizes $n_{i}=32,i\in\overline{1,10}$. The performance of the algorithm for different numbers of shared Tucker factors is summarized in Figure 10. This steps-like morphology shares its origin with that of Figure 9, stemming from the same iterative rank-incrementing process. Table 2 demonstrates relative differences between relative L$2$ error without using RStGD and with it.

These experiments demonstrate that despite the quasi-optimal nature of Algorithm 1, the returned approximation closely approaches local optima in terms of the Frobenius norm. Furthermore, the subsequent steepest gradient descent step yields only marginal improvements in relative error in both tested configurations.

7.2 Eigenvalue problem

This section presents numerical solutions for the eigenvalue problem of symmetric positive definite matrices, constrained to the manifold of fixed SF-ETT rank. We consider two test cases: a discretized Laplace operator with and without the Henon-Heiles potential matrix [6]. Our approach utilizes a Riemannian optimization framework. Similar and other Riemannian approaches for the eigenproblem can be found in [3, 36, 37, 18].

Computing the extreme eigenvectors and eigenvalues of a symmetric positive definite matrix $H$ is equivalent to minimizing the Rayleigh quotient:

[TABLE]

We employ the Locally Optimal Block Preconditioned Conjugate Gradient (LOBPCG) algorithm [17] for a single vector. In the SF-ETT case we replace the whole search space $\mathbb{R}^{n_{1}\times\dotsm\times n_{d}}$ with the SF-ETT manifold of fixed rank.

The LOBPCG algorithm is a gradient-based method for solving eigenvalue problems. Unlike simple gradient descent, it leverages not only the gradient at the current iterate but also information from gradients computed in previous iterations. At each step, the algorithm constructs a search subspace spanned by the current eigenvector approximation, the current gradient, and a set of accumulated search directions. A locally optimal solution within this subspace is then found using the Rayleigh-Ritz procedure $\mathtt{RayleighRitz}\left(H,S\right)$. In particular, let $S$ be a basis for a search space. Then we need to solve the following generalized eigenvalue problem:

[TABLE]

Although LOBPCG is capable of computing multiple eigenpairs simultaneously (block method), in the present experiments it is configured to calculate only the eigenpair corresponding to the smallest eigenvalue.

We use a Riemannian version of the Locally Optimal Conjugate Gradient (LOCG) Algorithm 2 that is constrained to the manifold of fixed SF-ETT rank. Preconditioning and block versions could also be included, but we restrict ourselves to the basic case to isolate the effect of sharing. Algorithm 2 approximates the unit eigenvector, which corresponds to the smallest eigenvalue. In practice, this approximation is stored as $(d-1)$-orthogonal SF-ETT tensor, where $d$ is the amount of TT cores and Tucker factors.

7.2.1 Laplace operator

The eigenproblem for the discretized Laplace operator on a uniform grid, which arises when searching for the states of an isolated particle in a $d$-dimensional box, can be written as follows:

[TABLE]

where $I_{n}\in\mathbb{R}^{n\times n}$ is an identity matrix and $D=\mathrm{tridiag}\,\left(1,-2,1\right)\in\mathbb{R}^{n\times n}$. The tensor $L$ has a maximum TT rank bounded by $2$ [27, Proposition $4.1$].

The analytical solution to this problem is known in advance, so the task primarily serves as a benchmark for eigensolvers. The eigenvalues of the matrix $(-D)$ are formed in pairs ${\theta_{l},X_{l}}$:

[TABLE]

The performance of Algorithm 2 for the Laplace operator with different amount of shared Tucker factors and different ranks are pictured in Figure 11. Figure 12 demonstrates the reduction in computation time achieved by Algorithm 2 with an increasing number of shared Tucker factors.

7.2.2 Henon-Heiles

Another setup is an eigenproblem given by the following matrix

[TABLE]

where $L$ is the discretized Laplace operator as in Section 7.2.1 and $\hat{V}$ denotes a tensor of discretized Henon-Heiles potential function $V$ on a uniform grid:

[TABLE]

The tensor $\hat{V}$, representing the Henon-Heiles potential, has a maximum TT rank bounded by 3 [27, Lemma $4.10$]. To get a proper SF-ETT approximation of the Henon-Heiles potential we utilize the same approach as in Section 7.1 without RStGD procedure. One may notice that utilizing of such approximation for (25) depends on the number of shared Tucker factors since SF-ETT approximation of the Henon-Heiles potential (26) depends on it.

The experiment consists of approximating the Henon-Heiles potential (26) and constructing the bilinear function (25), subsequently computing its minimal eigenpair using Algorithm 2. For validation, this eigenpair was also calculated using the direct eigh solver from the ttpy package [31], which serves as a reference.

The performance of Algorithm 2 applied to (25) is shown in Figure 13 for varying numbers of shared Tucker factors and different ranks. Figure 14 demonstrates the corresponding reduction in iteration time as the number of shared Tucker factors increases. As the computations for (25) are more complex, the increased number of shared factors yields a more notable speedup than in the case of the simple Laplace operator (see Figure 12).

8 Conclusion

We explore a Riemannian optimization approach for an ETT decomposition with an additional constraint that enforces equality of the factors. We develop a general framework, along with its software implementation, that provides the necessary components for Riemannian optimization algorithms. This framework can be readily applied to achieve further memory and time reduction where the classical ETT decomposition is of interest.

\bmhead

Acknowledgements

The authors are thankful to Danil Kolyadin for valuable consultations concerning the rounding algorithm.

Appendix A Projection on tangent space of SF-ETT manifold

Proposition 7** (Tangent space structure).**

Let $\mathcal{X}\in\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})$ is a tangent foot with SF-ETT decomposition

[TABLE]

then the tangent space $T_{\mathcal{X}}\mathrm{SFETT}$ has the following structure

[TABLE]

where

[TABLE]

Proof.

Let $\xi\in L_{0}^{(i)},\eta\in L_{j},\forall i\in\overline{1,d},j\in\overline{1,d_{t}}$ then

[TABLE]

due to the gauge condition

[TABLE]

Now let $\eta\in L_{s}$ then

[TABLE]

due to the gauge condition

[TABLE]

Finally, let $\xi\in L_{0}^{(i^{\prime})},i\neq i^{\prime}$ then

[TABLE]

due to the gauge condition

[TABLE]

Thus $\forall i\in\overline{1,d},i^{\prime}\in\overline{1,d},i^{\prime}\neq i,\forall j\in\overline{1,d_{t}}$ it is implied that $L_{0}^{(i)}\perp L_{0}^{(i^{\prime})}$, $L_{0}^{(i)}\perp L_{j}$, $L_{0}^{(i)}\perp L_{s}$. The orthogonality of $L_{j}$ against $L_{j^{\prime}}$ and against $L_{s}$ is the same as in [34]. ∎

Proposition 8** (Tangent space projection).**

Let $Z\in\mathbb{R}^{n_{1}\times\dotsc\times n_{d_{t}}\times n_{s}\times\dotsc\times n_{s}}$ and $\mathcal{X}\in\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})$ is a tangent foot with SF-ETT decomposition

[TABLE]

then $P_{T_{\mathcal{X}}\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})}\left(Z\right)\in T_{\mathcal{X}}\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})$ has SF-ETT decomposition as follows

[TABLE]

Proof.

According to Proposition 7 every $P_{T_{\mathcal{X}}\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})}\left(Z\right)\in T_{\mathcal{X}}\mathrm{SFETT}$ has a unique representation as sum of elements of $L_{0}^{(i)},L_{i},L_{s}$

[TABLE]

where components can be obtained via orthogonal projectors

[TABLE]

If $Z\in\mathbb{R}^{n_{1}\times\dotsc\times n_{d_{t}}\times n_{s}\times\dotsc\times n_{s}}$ then then projection formulas for $\dot{V}^{(i)}$ and $\dot{U}$ are the same as in [34] since $\mathcal{X}$ can be seen as element of $\mathrm{SFT}\,(\mathbf{r}^{ts})$.

So one need only to find out $P_{L_{0}^{(i)}}Z$. Since $P_{L_{0}^{(i)}}$ is an orthogonal projector for any $i\in\overline{1,d}$ it satisfies

[TABLE]

Let $i=d$ then

[TABLE]

So the scalar product at the left hand side due to the orthogonality of Tucker factors is equal to

[TABLE]

The scalar product at the right hand side equals to

[TABLE]

Thus the following identity is obtained

[TABLE]

Due to the in [22, Theorem $3.1$] one can obtain

[TABLE]

Let $i\in\overline{1,d-1}$ then

[TABLE]

Left hand side scalar product is equal to

[TABLE]

Right hand side equals to

[TABLE]

Thus the following holds

[TABLE]

Due to the [22, Theorem $3.1$] one can obtain

[TABLE]

∎

Appendix B Riemannian Autodiff

Proposition 9** (Partial derivatives).**

Let $\mathcal{X}\in\mathrm{SFETT}\,(\mathbf{r}^{tt},\mathbf{r}^{ts})$ and

[TABLE]

Let also $\mathcal{C}$ operator be a following operator

[TABLE]

Let $g$ be a scalar function defined as

[TABLE]

where $f:\mathbb{R}^{n_{1}\times\dots n_{d_{t}}\times n_{s}\times\dots\times n_{s}}\to\mathbb{R}$ is an arbitrary smooth function defined in some neighborhood of point $\mathcal{X}$. Then

[TABLE]

Proof.

The partial derivatives with respect to $B^{(i)},i\in\overline{1,d_{t}}$ and $A$ are the same as in the case of SF-Tucker decomposition [34]. So one only need to check the equalities above for the case of partial derivatives with respect to $S^{(i)},i\in\overline{1,d}$.

Now let $i\in\overline{1,d-1}$

[TABLE]

Now applying vectorization operation on both of the vectors inside the scalar product lefts the result unchanged so

[TABLE]

For $i=d$ one can perform the same operations. ∎