Generative Tensor Network Classification Model for Supervised Machine Learning

TL;DR

The paper introduces a generative tensor network classification model for supervised learning that achieves competitive accuracy on image datasets, demonstrating efficiency and natural data clustering in quantum-inspired Hilbert space.

Contribution

It proposes a novel generative tensor network approach for classification, outperforming some traditional models and providing insights into data clustering in Hilbert space.

Findings

Achieves high accuracy on MNIST and Fashion-MNIST datasets.

GTNC outperforms naive Bayes and SVM classifiers.

Efficiently captures data structure with finite bond dimensions.

Abstract

Tensor network (TN) has recently triggered extensive interests in developing machine-learning models in quantum many-body Hilbert space. Here we purpose a generative TN classification (GTNC) approach for supervised learning. The strategy is to train the generative TN for each class of the samples to construct the classifiers. The classification is implemented by comparing the distance in the many-body Hilbert space. The numerical experiments by GTNC show impressive performance on the MNIST and Fashion-MNIST dataset. The testing accuracy is competitive to the state-of-the-art convolutional neural network while higher than the naive Bayes classifier (a generative classifier) and support vector machine. Moreover, GTNC is more efficient than the existing TN models that are in general discriminative. By investigating the distances in the many-body Hilbert space, we find that (a) the samples are naturally clustering in such a space; and (b) bounding the bond dimensions of the TN's to finite values corresponds to removing redundant information in the image recognition. These two characters make GTNC an adaptive and universal model of excellent performance.

Figures

Equations

$${\mathbf{s}}_i={\left[\cos \left(\frac{\pi }{2}{x}_i\right),\sin \left(\frac{\pi }{2}{x}_i\right)\right]}^T.$$

$$f_c=|{\mathbf{v}}^{†}{\mathbf{\Psi }}^{(c)}|,$$

$${\tilde{\mathbf{\Psi }}}^{(c)}=\sum _{\mathbf{v}\in c}\mathbf{v}/\sqrt{N_c},$$

$$D_{c_1{c}_2}=\frac{1}{N_{c_1}{N}_{c_2}}\sum _{x\in c_1}\sum _{y\in c_2}\sqrt{\sum _{i=1}^L{\left(x_i-y_i\right)}^2},$$

$$F_{c_1{c}_2}=\frac{{\sum }_{\mathbf{u}\in c_1}{\sum }_{\mathbf{v}\in c_2}{\mathbf{u}}^{†}\mathbf{v}}{\sqrt{N_{c_1}{N}_{c_2}}}.$$

$${\Psi }_{s_1{s}_2\cdots s_L}=\sum _{{\alpha }_1\cdots {\alpha }_{L-1}}{T}_{s_1,{\alpha }_1}^{[1]}{T}_{s_2,{\alpha }_1{\alpha }_2}^{[2]}\cdots T_{s_{L-1},{\alpha }_{L-2}{\alpha }_{L-1}}^{[L-1]}{T}_{s_L,{\alpha }_{L-1}}^{[L]}.$$

$${\mathbf{T}}^{[l]}\leftarrow {\mathbf{T}}^{[l]}-\eta \frac{\partial \Gamma }{\partial {\mathbf{T}}^{[l]}},$$

$$\Gamma =-\frac{1}{J}\sum _j\ln \frac{P_j}{Z}-\ln \left(J\right).$$

$$\Gamma =\sum _j{\left|{\tilde{L}}_j-L_j^c\right|}^2.$$

Full text

I Introduction

Machine learning incorporating with the principles of quantum mechanics forms a novel interdisciplinary field known as quantum machine learning Biamonte et al. (2017). Among many sub-directions, machine learning in quantum space is currently under hot debate. The quantum space, also called Hilbert space or quantum-enhanced feature space, is where the quantum states and operators live. The quantum classifiers defined in such a space have been proposed, which are expected to work on quantum hardwares such as superconducting processors Havlicek et al. (2019); Schuld and Killoran (2019).

In recent years, booming progresses have been made by combining quantum physics and machine learning through tensor network (TN) Cichocki et al. (2016, 2017); Glasser et al. (2018). TN is a powerful tool that originates from quantum many-body physics and quantum information sciences; it can be applied to efficiently deal with the states and operators defined in many-body Hilbert space whose dimension increases exponentially with the number of sites (or physical particles) Verstraete et al. (2008); Orús (2014); Ran et al. (2017); Evenbly and Vidal (2011); Bridgeman and Chubb (2017); Schollwöck (2011); Cirac and Verstraete (2009a). As a novel extension, TN is considered as a universal model for supervised and unsupervised learning Stoudenmire and Schwab (2016); Liu et al. (2017, 2018); Han et al. (2018); Cheng et al. (2018); Pestun and Vlassopoulos (2017); Cheng et al. (2019); Chen et al. (2018); Guo et al. (2018). Its applications on, e.g., image recognition, already exhibit competitive performance to the conventional models such as neural networks.

With the underlying connections between TN and quantum circuits Kak (1995); Schuld and Killoran (2019); Benedetti et al. (2018); Mitarai et al. (2018); Rebentrost et al. (2014); Lloyd et al. (2013); Liu and Wang (2018); Zeng et al. (2018); McClean et al. (2016); Farhi et al. (2014); Farhi and Neven (2018), TN sheds new lights on quantum computation of machine learning tasks Schuld et al. (2015); Arunachalam and de Wolf (2017); Wiebe et al. (2012). For instance, a training algorithm Liu et al. (2017) inspired by the multiscale entanglement renormalization ansatz Cincio et al. (2008) allows using unitary gates or isometries to construct the TN for machine learning. Most recently, Huggins et al. proposed that the quantum circuit corresponding to a TN can be easily designed by one- and two-qubit unitary gates Huggins et al. (2019). The appealing perspective of TN in quantum machine learning urges us to understand deeply the underlying characters of the TN machine learning, and to develop novel TN approaches of higher performance. However, some fundamental questions are still untouched. Among others, it is elusive about possible advantages of TN machine learning in quantum many-body space, compared with the models (e.g., neural network) that learn the data in the original multiple-scalar space.

In this work, we propose a generative TN classification (GTNC) model for supervised learning (Fig.1) in many-body Hilbert space (denoted as $\mathcal{H}$). The GTNC is formed by several generative TN’s; each generative TN is a quantum state defined in $\mathcal{H}$ and is trained as the generative model for the corresponding class of images Han et al. (2018). For a given sample, the classification is done by finding the generative TN with the smallest Euclidean distance (fidelity) to this sample in $\mathcal{H}$. In other words, the classification is given by the boundary, from which the Euclidean distances to the generative TN’s are equal. With the MNIST Deng (2012) and fashion-MNIST Xiao et al. (2017) datasets, the GTNC shows remarkable efficiency and accuracy by comparing with several existing methods including the discriminative TN machine learning method Stoudenmire and Schwab (2016), supportive vector machines (SVM’s) Cortes and Vapnik (1995), and naive Bayes classifiers Rish et al. (2001).

Two key advantageous characters of GTNC are discussed. Firstly by computing the Euclidean distances (i.e., fidelity) among the samples and the generative TN’s, we show that the samples mapped to the many-body Hilbert space are naturally clustering. It implies that the classification can be efficiently and accurately done in such a space. The clustering should be an advantage from the space $\mathcal{H}$. Though the idea of mapping to such a space of a much higher dimension is analogous to SVM, better accuracy is achieved with GTNC. Secondly by comparing with a lazy-learning baseline model, we show that bounding the bond dimensions of the TN’s to finite values corresponds to removing redundant information in the image recognition. The relation to the quantum entanglement can be addressed.

II Generative tensor network classification algorithm for supervised learning

The training of GTNC is to obtain the generative TN $\mathbf{\Psi}^{(c)}$ ($c=1,\cdots,K$ with $K$ the total number of classes in the classification task) for each of the classes. We use the algorithm proposed by Han et al. Han et al. (2018). To begin with, one builds a one-to-one map called feature map Stoudenmire and Schwab (2016); Novikov et al. (2016), which maps the images to a vector space known as many-body Hilbert space (denoted as $\mathcal{H}$) in quantum physics. For example, the feature map that transforms the $i$-th pixel $x_{i}$ (normalized so that $0\leq{x_{i}}\leq 1$) to a two-component vector can be written as

[TABLE]

In this way, one image that consists of $L$ pixels is mapped to the direct product of $L$ vectors as $\mathbf{v}=\prod_{\otimes i}{{\mathbf{s}_{i}}}$. Physically, $\mathbf{v}$ can be regarded as the product state of $L$ qubits. Each qubit has two components, equivalent to a spin-$1/2$. Note that it is possible to generalize the feature map to be $d$-component with $d>2$. Then one image is mapped to a vector defined in the $d^{L}$-dimensional vector space.

TN is utilized as the generative model of each class Verstraete et al. (2008); Cirac and Verstraete (2009b); Schollwöck (2011); Orús (2014). In fact, the generative models are quantum states of $L$ bodies defined in $\mathcal{H}$, which capture the joint probability distributions of the corresponding sets. In quantum many-body physics, TN has been shown as an efficient and power tool to deal with quantum many-body states, where the computational complexity can be reduced from exponentially-hard to polynomial-hard. In Ref. Han et al. (2018), the generative TN is trained with a gradient algorithm that minimizes the cost function of Kullback–Leibler divergence Kullback and Leibler (1951).

After training the generative TN’s $\{\mathbf{\Psi}^{(c)}\}$, a given sample can be classified by comparing the Euclidean distances in $\mathcal{H}$ between this sample and $\{\mathbf{\Psi}^{(c)}\}$. We choose the fidelity $f_{c}$ to measure the distance, which is defined as

[TABLE]

with $\mathbf{v}$ the sample after the feature map. Note that in quantum information, fidelity is a measurement of distance between two quantum states. The classification is indicated by finding the largest fidelity, i.e., $\arg\max_{c}f_{c}$. One can find the pseudo code of GTNC in Sec.V.1.

III Experiments

III.1 GTNC: an adaptive generative classification model

On the MNIST dataset, GTNC is compared with other well-established methods (Fig.2), i.e., classical generative classifiers (naive Bayes classifiers), high-dimensional classifiers (SVM’s) and a baseline model which is a lazy-learning model using feature map without TN (see Eq.3). For GTNC, different bond dimensions $\chi$ are taken, which controls the number of variational parameters (see Sec.V.1 for details). We also testify on the fashion-MNIST. The 10-class testing accuracy of GTNC reaches around $88.2\%$, while for the SVM’s it varies from $48.4\%$ to $89.7\%$ depending on the parameters Xiao et al. (2017). For the naive Bayes classifiers, the testing accuracy is no higher than $70.6\%$ for the fashion-MNIST dataset.

The experiments given by Fig. 2 reveal several advantages of GTNC. One is that the accuracy of GTNC significantly surpasses the naive Bayes classifiers. Note that it is usual to use the discriminative models to do classification, such as convolutional neural networks (CNN’s). The state-of-the-art accuracy is $99.77\%$ Ciresan et al. (2012) for MNIST and $92.54\%$ Bhatnagar et al. (2017) fashion-MNIST, respectively. It is true that the accuracy of GTNC is competitive but still lower than the best discriminative models. Nevertheless, The previous knowledge is necessary for these discriminative models (such as architecture and other hyper-parameters that can largely affect the results) to reach the best accuracy. For GTNC in contrast, we use the same architecture of the TN (a 1D TN that is the same as Ref. Han et al. (2018)) and the same hyper-parameters (such as the feature map) for different datasets. We are optimistic to improve further the accuracy of GTNC by optimizing the architecture and hyper-parameters, which however is beyond the scope of the current work.

Secondly, GTNC possesses striking resemblance as well as essential differences compared with the SVM’s. The main idea of SVM is to map the samples to a much higher dimensional space, where it becomes relatively easy to find the boundary for classification. For GTNC, the feature map is to map the samples to the many-body Hilbert space that is exponentially large. By training the generative TN’s in such a space, the boundary for classification is found by computing the fidelity.

It is the underlying difference that makes GTNC superior than SVM. The first difference concerns the kernel function and the space. In most cases of SVM, the mapping method is implicit and determined by a positive-defined matrix which is the distance in the higher dimensional space. This distance matrix is calculated by a certain kernel function like radial basis function kernel with origin data. Mercer’s condition can guarantee that the kernel function will correspond to a higher dimensional space Mercer and Forsyth (1909). The mapping method is implicit, thus it becomes extremely challenging to analyse how to improve the performance of SVM. The results strongly depend on the space to where the data are mapped, and the hyper-parameters such as the soft margin Cortes and Vapnik (1995). There is no general theories of finding the best parameters of SVM Leslie et al. (2001), which hinders the applications of SVM to new challenging problems.

In comparison, GTNC is more universal and less parameter-dependent. The kernel function in GTNC is determined by the feature map and can be explicitly written, which satisfies Mercer’s condition. For different datasets, we use the same feature map [see Eq. (1)] to transform the data to the higher-dimensional space. It is possible to optimize the kernel function of GTNC to further improve the performance.

The general strategies of GTNC and SVM are also different. The GTNC is formed by several generative models, each of which learns the joint probability distribution of one class of samples. The classification is determined by taking the generative models as the references. Such a strategy works well due to the clustering of the samples in $\mathcal{H}$, giving higher accuracy than the naive Bayes classifiers. It is also avoided to input the samples of all classes at the same time to train the classifier(s), which leads to higher efficiency compared with the discriminative algorithms. For SVM, it is to find the classification boundary in the higher-dimensional space. This might also be one reason that the results of SVM largely depends on the chosen space and the hyper-parameters.

We shall note when we build a SVM model with the kernel function from the feature map, the testing accuracy is extremely poor (no more than $30.0\%$). This suggests that the kernel from the feature map works with the algorithms of SVM much worse than the generative TN algorithm Han et al. (2018).

Thirdly, the accuracy of GTNC also surpasses the baseline model I with a moderate bond dimension ($1\ll\chi\ll d^{L}$ with $d^{L}$ the dimension of $\mathcal{H}$). The baseline model I is a lazy-learning version of GTNC; the generative model $\tilde{\mathbf{\Psi}}$ of the $c$-th class can be defined as

[TABLE]

with $N_{c}$ the number of samples in the $c$-th class. It means ${\widetilde{\bf{\Psi}}^{(c)}}$ is simply the summation of all vectorized samples in the $c$-th class.

For classifying a given sample $\mathbf{v}$, we still use Eq. (2) to define the fidelity as $\tilde{f}_{c}=|\mathbf{v}^{\dagger}\tilde{\mathbf{\Psi}}^{(c)}|$. The classification of $\mathbf{v}$ is given by $\arg\max_{c}\tilde{f}_{c}$. Different from GTNC, we do not need to train $\tilde{\mathbf{\Psi}}$ to classify. The fidelity can be directly calculated as $\tilde{f}_{c}=|\sum_{\mathbf{u}\in c}\mathbf{v}^{\dagger}\mathbf{u}|/\sqrt{N_{c}}$, which makes baseline model I a lazy-learning model.

Let us consider to write $\tilde{\mathbf{\Psi}}$ in a TN form just like $\mathbf{\Psi}$ in GTNC. It is expected that the bond dimension of $\tilde{\mathbf{\Psi}}$ should be extremely large. In other words, $\mathbf{\Psi}$ in GTNC can be understood as a finite-bond-dimensional approximation of $\tilde{\mathbf{\Psi}}$. Surprisingly, the accuracy of GTNC is higher than the baseline model I. It implies that by taking a moderate bond dimension, some redundant information are removed and a better classification can then be made.

The value of $\chi$ actually characterizes the capacity of quantum entanglement that the TN (state) can carry. The entanglement entropy of the TN here is the Rényi entropy of the dataset, which is defined as ${H_{\alpha}}={1\over{1-\alpha}}\sum\limits_{i=1}^{n}{\log\left({p_{i}^{\alpha}}\right)}$ with ${p_{i}}$ the probabilities to have the $i$-th sample and $\alpha$ a constant Rényi et al. (1961). The entanglement entropy corresponds to the case of $\alpha=2$. The Rényi entropy satisfies ${H_{2}}\left(\chi\right)\leq\log\left(\chi\right)$. In other words, by reducing $\chi$, the maximum of Rényi entropy becomes smaller. The regularization process in GTNC (known as canonicalization Orús and Vidal (2008)) guarantees that one always discards the less entangled basis. A former work showed that the less-entangled sites (pixels) contain less-important information, which can be discarded without harming too much the accuracy Liu et al. (2018) It means the TN machine learning can be implemented more efficiently with a much smaller number of features. In accordance, our experiments demonstrate that the important information is restored in the highly-entangled basis. It is suggested that with the same number of features, the number of variational parameters in the TN can be safely reduced by removing the less-entangled basis. This shows that the over-fitting of the TN machine learning can be avoided in a controllable manner according to the quantum entanglement.

III.2 Natural clustering

To further understand the GTNC, we calculate the distances of the training samples in different spaces (Fig.3). The Euclidean distance between the $c_{1}$ and $c_{2}$-th classes in original multi-scalar space is defined as

[TABLE]

where the pixels are normalized as $0\leq x\leq 1$. In the many-body Hilbert space $\mathcal{H}$, the fidelity is used to represent distance of two classes, which is defined as

[TABLE]

$F_{{c_{1}}{c_{2}}}$ characterizes the closeness of two classes of images.

In the original space, the distances are at the same order of magnitude for the samples of the same class or of two different classes [Fig.3 (a)]. It means the distribution of the samples in this space is more or less random. In many-body Hilbert space $\mathcal{H}$, the fidelity in $\mathcal{H}$ between different classes is over $10^{5}$ times lower than those with the same label (the diagonal terms $F_{cc}\simeq 1$) [Fig.3 (b)]. In other words, the distance between the samples of different classes is averagely much larger than that between the samples of a same class. This means the samples of the same class are clustering in $\mathcal{H}$, which makes it much easier to classify.

The clustering is also consistent with the fair accuracy of the baseline model I. Let us rewrite $F_{{c_{1}}{c_{2}}}$ in terms of the generative TN’s of the baseline model I [Eq. (3)]. We have ${F_{{c_{1}}{c_{2}}}}=|{{\bf{\tilde{\Psi}}}^{({c_{1}}){\dagger}}}{{\bf{\tilde{\Psi}}}^{({c_{2}})}}|$, which is the fidelity of the two generative TN’s. As the samples are clustering, the distance from a given sample to the correct ${\bf{\tilde{\Psi}}}^{({c})}$ should be much smaller than that to a wrong one. Thus, the classification can be accurately done by comparing the distances. Note the Euclidean distance in $\mathcal{H}$ can be deducted from fidelity, satisfying $D_{c_{1}c_{2}}=|\mathbf{\tilde{\Psi}}^{(c1)}-\mathbf{\tilde{\Psi}}^{(c2)}|^{2}=F_{c_{1}c_{1}}^{2}+F_{c_{2}c_{2}}^{2}-2F_{{c_{1}}{c_{2}}}$ with $F_{cc}=|\mathbf{\tilde{\Psi}}_{c}|$. We do not enforce the normalization of $\{\mathbf{\tilde{\Psi}}^{(c)}\}$, though we have $|\mathbf{\tilde{\Psi}}_{c}|\simeq 1$, giving $D_{c_{1}c_{2}}\simeq 2-2F_{{c_{1}}{c_{2}}}$.

We also compare GTNC with th existing discriminative TN model (dubbed as the baseline model II) Stoudenmire and Schwab (2016). The pseudo-code can be found in the Sec.V.2. Our experiments show that the efficiency of GTNC is significantly higher than the baseline model II. On MNIST dataset with $\chi=32$, the accuracy of GTNC converges to $97.6\%$ in about $9\times{10^{3}}$ seconds of CPU time, while the accuracy of the baseline model II converges to $87.3\%$ in about $4\times{10^{6}}$ seconds 111The CPU model is Intel(R) Xeon(R) CPU E5-2630 v3 @ 2.40GHz. . The efficiency differs due to the strategy. For GTNC, one will only input one class of images to train each of the generative TN, and the tensors converge with a small number of iterations. For the baseline model II, one will input the samples of all classes to train the classifier, and it needs much more iterations to converge. The complexity analysis shows that even in one iteration, the computational complexity of GTNC is much lower than baseline model II because the samples only need to be input into the corresponding tensor network in GTNC.

IV Discussion and Perspective

In this work, we propose the generative TN classification (GTNC) method, and based on it investigate several fundamental issues of the TN machine learning, i.e., the roles played by the feature map and by the bond dimensions of the TN representation. The main contributions of this work are concluded in the following.

• •

GTNC is proposed as a generative model for supervised machine learning. The central idea is to individually train the generative TN’s in many-body Hilbert space for samples with different labels, and to classify by comparing the distances. The performance of GTNC surpasses the existing (discriminative) TN-machine learning methods, the Naive Bayes method which are also generative classifier, and the supportive vector machine.

• •

The role of feature map is revealed. We show that the feature map of the TN machine learning methods is to map the samples to an exponentially large vector space (called many-body Hilbert space in physics). In such a space, the samples are naturally clustering, where the classification can be easily and accurately done with the help of the generative TN’s.

• •

The relation between entanglement and machine learning is discussed, which is useful to avoid over-fitting in a controllable way. The experiments by comparing GTNC with baseline model I imply that the important information is restored in the highly-entangled basis of the generative TN’s. By keeping a proper number of the relatively highly-entangled basis, the accuracy surpasses the baseline model I, where all bases are taken into consideration.

Our work contributes to answering an important question: whether there exist any advantages to solve machine learning problems in the exponentially-large many-body Hilbert space by TN than in the multiple-scalar space by the classical machine learning models. While the previous simulations of the TN machine learning algorithms have given considerable promising results, our experiments show a positive answer in a more explicit way. Such investigations will strongly motivate to develop the quantum computation of machine learning in the many-body Hilbert space, such as the machine learning schemes by quantum circuits Huggins et al. (2019). The benefits or “quantum supremacy” will be not just limited to quantum acceleration, but also to develop more universal, powerful, and well-controlled machine learning models.

V Methods

V.1 Tensor network machine

The functions (vectors or operators) defined in this exponentially large space $[\mathbb{V}^{d}]^{\otimes L}$ might have lager potential for generating and learning compared with the multi-scalar functions (such as neural network). One problem is how to handle such an exponentially large space, which is usually NP-hard by classical computers.

The quantum many-body physics provides us a solution called tensor network (TN) , which reduces the cost from exponential to polynomial or linear manner Verstraete et al. (2008); Cirac and Verstraete (2009b); Schollwöck (2011); Orús (2014). TN is an efficient representation of one (large) tensor by writing it as the contraction of several tensors. Matrix product state Perez-Garcia et al. (2006) is one form of the TN’s (Fig.4), which has been utilized in the machine learning field Han et al. (2018); Stoudenmire and Schwab (2016). An MPS formed by $L$ tensors can be written as

[TABLE]

$T^{[l]}$ is a $(d\times\chi\times\chi)$-dimensional tensor located on the $l$-th site, the indexes $\{\alpha_{l}\}$ and $\{s_{n}\}$ ($l=1,\cdots,L-1$) are dubbed as virtual and physical bonds, respectively. $\chi$ is called the bond dimension of the MPS, which determines the number of parameters and the upper bound of the entanglement that the MPS can capture (see for example Ref. Tagliacozzo et al. (2008)). For simplicity, we assume all elements of the tensors are real numbers. It is easy to see that by contracting all the virtual bonds, $\mathbf{\Psi}$ is a vector in the $[\mathbb{V}^{d}]^{\otimes L}$ space, whose dimension increases exponentially with $L$. Thanks to the TN structure of $\mathbf{\Psi}$, the number of parameters is about $Ld\chi^{2}$ in the MPS, which scales only linearly with $L$.

The gradient descent algorithm is used to optimize the MPS. For example, the $l$-th tensor of the MPS is updated as

[TABLE]

where $\eta$ is the step of the gradient decent algorithm and $\Gamma$ is the cost function. All tensors are updated iteratively until the preset convergence is reached. One could refer to Refs. Han et al. (2018); Stoudenmire and Schwab (2016) for more details.

V.2 Pseudo code of GTNC and baseline model II

In the GTNC, the algorithm for training the generative MPS’s follows Ref. Han et al. (2018). The baseline modelI algorithm follows Ref. Stoudenmire and Schwab (2016). And the pseudo codes are shown in Al.V.2 and Al.V.2.

Algorithm 1 - GTNC

1:$\alpha$:Step size;

2:$\beta$:decay rate for Step size;

3:$t\leftarrow 0$ (Initialize time step) $\Gamma\left({{T^{\left[1\right]}},{T^{\left[2\right]}},\cdots,{T^{\left[L\right]}}}\right)$: Stochastic objective function with parameters${\left\{{{T^{\left[l\right]}}}\right\}}$ [a]

4:$\left\{{T_{0}^{\left[l\right]}}\right\}$: Initial parameter vector$\left\{{{p_{l,0}}}\right\}\leftarrow 0$ (Initialize moment vector)

5:while $\left\{{T_{t}^{\left[l\right]}}\right\}$not converged do

6:     $t\leftarrow t+1$

7:     for $l=1:L$ do

8:         ${g_{l,t}}\leftarrow{\left.{{{d{\Gamma_{t}}}\over{d{T^{\left[l\right]}}}}}\right|_{{T^{\left[l\right]}}=T_{t-1}^{\left[l\right]}}}$

9:         ${p_{l,t}}\leftarrow{{g_{l,t}^{2}}\over{{{\left|{T_{t-1}^{\left[l\right]}}\right|}^{2}}}}$

10:         $T_{t}^{\left[l\right]}\leftarrow T_{t-1}^{\left[l\right]}-\alpha\cdot{{{g_{l,t}}}\over{\sqrt{{p_{l,t}}}}}$

11:         if $l<L$ then

12:              $Q,R\leftarrow qr\left({T_{t}^{\left[l\right]}}\right)$($Q$ is an unitary matrix matrixes. $R$ is an upper triangular matrix. And they satisfy $QR=T_{t}^{\left[l\right]}$)

13:              $T_{t}^{\left[l\right]}\leftarrow Q,T_{t}^{\left[{l+1}\right]}\leftarrow RT_{t-1}^{\left[{l+1}\right]}$

14:         end if

15:     end for

16:     if $\Gamma\left({\left\{{T_{t}^{\left[l\right]}}\right\}}\right)>\Gamma\left({\left\{{T_{t-1}^{\left[l\right]}}\right\}}\right)$ then

17:         $\alpha\leftarrow{\alpha\mathord{\left/{\vphantom{\alpha\beta}}\right.\kern-1.2pt}\beta}$

18:     end if

19:end while

20:return $\left\{{T_{t}^{\left[l\right]}}\right\}$

Algorithm 2 - baseline model II

1:$\alpha$:Step size;

2:$\beta$:decay rate for Step size;

3:$t\leftarrow 0$ (Initialize time step) $\Gamma\left({{T^{\left[1\right]}},{T^{\left[2\right]}},\cdots,{T^{\left[L\right]}}}\right)$: Stochastic objective function with parameters${\left\{{{T^{\left[l\right]}}}\right\}}$ [b]

4:$\left\{{T_{0}^{\left[l\right]}}\right\}$: Initial parameter vector$\left\{{{p_{l,0}}}\right\}\leftarrow 0$ (Initialize moment vector)

5:while $\left\{{T_{t}^{\left[l\right]}}\right\}$not converged do

6:     $t\leftarrow t+1$

7:     for $l=1:L-1$ do

8:         ${g_{l,t}}\leftarrow{\left.{{{d{\Gamma_{t}}}\over{d{T^{\left[{l,l+1}\right]}}}}}\right|_{{T^{\left[{l,l+1}\right]}}=T_{t-1}^{\left[{l,l+1}\right]}}}$

9:         ${p_{l,t}}\leftarrow{{g_{l,t}^{2}}\over{{{\left|{T_{t-1}^{\left[{l,l+1}\right]}}\right|}^{2}}}}$

10:         $T_{t}^{\left[{l,l+1}\right]}\leftarrow T_{t-1}^{\left[{l,l+1}\right]}-\alpha\cdot{{{g_{l,t}}}\over{\sqrt{{p_{l,t}}}}}$

11:         $U,L,V\leftarrow svd\left({\;T_{t}^{\left[{l,l+1}\right]}}\right)$($U$ and $V$ are unitary matrixes. $L$ is diagonal matrix. And they satisfy $UL{V^{\dagger}}=T_{t}^{\left[{l,l+1}\right]}$)

12:         $T_{t}^{\left[l\right]}\leftarrow U,T_{t}^{\left[{l+1}\right]}\leftarrow L{V^{\dagger}}$ [The process of moving label is shown in Fig. 5]

13:     end for

14:     if $\Gamma\left({\left\{{T_{t}^{\left[n\right]}}\right\}}\right)>\Gamma\left({\left\{{T_{t-1}^{\left[n\right]}}\right\}}\right)$ then

15:         $\alpha\leftarrow{\alpha\mathord{\left/{\vphantom{\alpha\beta}}\right.\kern-1.2pt}\beta}$

16:     end if

17:end while

18:return $\left\{{T_{t}^{\left[l\right]}}\right\}$

Note: [a] The cost function is chosen as follow

[TABLE]

Note: [b] The cost function is chosen as follow

[TABLE]

Acknowledgments

This work was supported in part by the National Natural Science Foundation of China (11834014 and 11474279), the National Key R$\&$D Program of China (2018YFA0305800), and the Strategic Priority Research Program of the Chinese Academy of Sciences (XDB28000000). S.J.R. is supported by Beijing Natural Science Foundation (1192005 and Z180013) and Foundation of Beijing Education Committees under Grants No. KZ201810028043.