Tucker Decomposition Network: Expressive Power and Comparison

TL;DR

This paper introduces a Tucker tensor decomposition-based deep neural network, demonstrating its superior expressive power over shallow networks and its effectiveness in image classification tasks.

Contribution

It develops a Tucker decomposition network and analyzes its expressive power, showing it outperforms shallow networks and other tensor networks.

Findings

Tucker network is more expressive than shallow networks.

Exponential nodes are needed in shallow networks to match Tucker network.

Experimental results show Tucker network's effectiveness in image classification.

Abstract

Deep neural networks have achieved a great success in solving many machine learning and computer vision problems. The main contribution of this paper is to develop a deep network based on Tucker tensor decomposition, and analyze its expressive power. It is shown that the expressiveness of Tucker network is more powerful than that of shallow network. In general, it is required to use an exponential number of nodes in a shallow network in order to represent a Tucker network. Experimental results are also given to compare the performance of the proposed Tucker network with hierarchical tensor network and shallow network, and demonstrate the usefulness of Tucker network in image classification problems.

Tables

Table 1. Table 1: The parameters setting of Fig. 5 in Tucker network, Shallow network and Hierarchical tensor network.

Network	Num of parameters	Num $M$ of representation function	Filter size	Strides size	Rank $J_i\left(i\in \left[N\right]\right)$
Tucker		10	14 $\times$ 23	14 $\times$ 5	2
HT	3478	14	14 $\times$14	14 $\times$ 14	8
Shallow		10	16 $\times$21	12 $\times$ 7	2
Tucker		12	14 $\times$ 17	14 $\times$ 11	3
HT	3834	18	14 $\times$14	14 $\times$ 14	3
Shallow		16	14 $\times$16	14 $\times$ 12	3
Tucker		12	14 $\times$ 15	14 $\times$ 13	4
HT	5300	12	16 $\times$26	12 $\times$ 2	4
Shallow		10	20 $\times$21	8 $\times$ 7	4
Tucker		11	14 $\times$ 14	14$\times$ 14	5
HT	8657	11	26 $\times$27	2 $\times$ 1	11
Shallow		17	20 $\times$23	8 $\times$ 5	10

Table 2. Table 2: The parameters setting of Fig. 7 in Tucker network, Shallow network and Hierarchical tensor network.

Network	Num of parameters	Num $M$ of representation  function	Filter size	Strides size	Rank $J_i\left(i\in \left[N\right]\right)$
Tucker		10	16 $\times$ 26	16 $\times$ 6	3
HT	13432	10	21$\times$21	11 $\times$11	3
Shallow		10	17 $\times$26	15 $\times$6	3
Tucker		10	16 $\times$ 31	16 $\times$1	4
HT	17626	22	16 $\times$16	16$\times$16	6
Shallow		10	20 $\times$29	12$\times$3	4
Tucker		20	16 $\times$ 18	16$\times$14	5
HT	23970	30	16 $\times$16	16$\times$16	6
Shallow		12	25 $\times$26	7$\times$6	19
Tucker		24	16 $\times$16	16$\times$16	6
HT	32016	12	28 $\times$31	4$\times$1	9
Shallow		20	17 $\times$31	15$\times$1	4
Tucker		31	16 $\times$ 17	16$\times$15	7
HT	50233	43	18 $\times$ 21	14$\times$11	7
Shallow		37	17 $\times$26	15$\times$6	7

Equations

$$\mathcal{A}=\sum _{z=1}^Z{\lambda }_z{\mathbf{a}}^{z,1}\circ {\mathbf{a}}^{z,2}\circ \cdots {\mathbf{a}}^{z,N},i.e.,A_{d_1,d_2,\cdots ,d_N}=\sum _{z=1}^Z{\lambda }_z{\mathbf{a}}_{d_1}^{z,1}{\mathbf{a}}_{d_2}^{z,2}\cdots {\mathbf{a}}_{d_N}^{z,N},$$

$$\mathcal{A}=\mathcal{G}{\times }_1{\mathbf{U}}^{(1)}{\times }_2{\mathbf{U}}^{(2)}\times \cdots {\times }_N{\mathbf{U}}^{(N)},i.e.,A_{d_1,\cdots ,d_N}=\sum _{j_1,\cdots ,j_N}{g}_{j_1,\cdots ,j_N}{U}_{d_1,j_1}^{(1)}\cdots U_{d_N,j_N}^{(N)},$$

$$\mathcal{A}=\sum _{j_1,j_2,\cdots ,j_N}{g}_{j_1,j_2,\cdots ,j_N}^y({\mathbf{u}}_{j_1}^{(1)}\circ {\mathbf{u}}_{j_2}^{(2)}\circ \cdots \circ {\mathbf{u}}_{j_N}^{(N)}),$$

$$\mathcal{A}$$

$${\varphi }^{l,t_l,{\gamma }_l}$$

$${\varphi }^{1,t_1,{\gamma }_1}$$

$$F_y({\mathbf{x}}_1,{\mathbf{x}}_2,\cdots ,{\mathbf{x}}_N)=\sum _{d_1,d_2,\cdots ,d_N}^M{\mathcal{A}}_{d_1,d_2,\cdots ,d_N}^y\prod _{i=1}^N{f}_{{\theta }_{d_i}}({\mathbf{x}}_i),$$

$$F_y=\sum _{z=1}^Z{\lambda }_z^y\prod _{i=1}^N\sum _{d=1}^M{\mathbf{a}}_d^{z,i}{f}_{{\theta }_d}({\mathbf{x}}^{\mathbf{i}}).$$

$${\mathcal{A}}^y=\sum _{j_1,j_2,\cdots ,j_N}{g}_{j_1,j_2,\cdots ,j_N}^y({\mathbf{u}}_{j_1}^{(1)}\circ {\mathbf{u}}_{j_2}^{(2)}\circ \cdots \circ {\mathbf{u}}_{j_N}^{(N)}).$$

$$F_y$$

$${\mathbf{v}}^{(i)}=U^{(i)T}{f}_{\theta }({\mathbf{x}}_i),\text{leavevmode}\text{leavevmode}i\in [N],$$

$$F_y=⟨{\mathcal{G}}^y,{\mathbf{v}}^{(1)}\circ {\mathbf{v}}^{(2)}\cdots \circ {\mathbf{v}}^{(N)}⟩,$$

$$\mathcal{A}$$

$${\varphi }^{1,1,{\gamma }_1}$$

$${\varphi }^{1,2,{\gamma }_1}$$

$$\Phi^{1,1}=\left(\begin{array}[]{ccc}vec(\phi^{1,1,1})&\cdots&vec(\phi^{1,1,r_{1}})\\ \end{array}\right),$$

$$\Phi^{1,2}=\left(\begin{array}[]{ccc}vec(\phi^{1,2,1})&\cdots&vec(\phi^{1,2,r_{1}})\\ \end{array}\right),$$

$${\Sigma }_1=diag(a_1^2,\cdots ,a_{r_1}^2).$$

$${\varphi }^{1,1,{\gamma }_1}={\mathbf{U}}^{0,1}{\Sigma }^{0,1,{\gamma }_1}({\mathbf{U}}^{0,2}{)}^T,{\varphi }^{1,2,{\gamma }_1}={\mathbf{U}}^{0,3}{\Sigma }^{0,2,{\gamma }_1}({\mathbf{U}}^{0,4}{)}^T,$$

$$vec({\varphi }^{1,1,{\gamma }_1})=({\mathbf{U}}^{0,2}\otimes {\mathbf{U}}^{0,1})vec({\Sigma }^{0,1,{\gamma }_1}),vec({\varphi }^{1,2,{\gamma }_1})=({\mathbf{U}}^{0,4}\otimes {\mathbf{U}}^{0,3})vec({\Sigma }^{0,2,{\gamma }_1}),{\gamma }_1\in [r_1].$$

$${\Phi }^{1,1}=({\mathbf{U}}^{0,2}\otimes {\mathbf{U}}^{0,1}){\mathbf{D}}_1,{\Phi }^{1,2}=({\mathbf{U}}^{0,4}\otimes {\mathbf{U}}^{0,3}){\mathbf{D}}_2,$$

$$\mathbf{A}=({\mathbf{U}}^{0,2}\otimes {\mathbf{U}}^{0,1}){\mathbf{D}}_1{\Sigma }_1{\mathbf{D}}_2^T({\mathbf{U}}^{0,4}\otimes {\mathbf{U}}^{0,3}{)}^T.$$

$$\mathcal{A}=\mathcal{G}{\times }_1{\mathbf{U}}^{0,2}{\times }_2{\mathbf{U}}^{0,1}{\times }_3{\mathbf{U}}^{0,3}{\times }_4{\mathbf{U}}^{0,4},$$

$$rank[\mathcal{A}{]}^{(p,q)}\le Z.$$

$$rank[\sum _{z=1}^Z{\lambda }_z^y{\mathbf{a}}^{z,1}\circ {\mathbf{a}}^{z,2}\circ \cdots \circ {\mathbf{a}}^{z,N}{]}^{(p,q)}$$

$$[A{]}^{(p,q)}=({\mathbf{U}}^{(p_1)}\otimes {\mathbf{U}}^{(p_2)}\otimes \cdots \otimes {\mathbf{U}}^{(p_{n_1})})[\mathcal{G}{]}^{(p,q)}({\mathbf{U}}^{(q_{n_2})}\otimes {\mathbf{U}}^{(q_{n_2-1})}\otimes \cdots \otimes {\mathbf{U}}^{(q_1)}{)}^T.$$

$$A_{d_1,\cdots ,d_N}=\sum _{j_1,\cdots ,j_N}{g}_{j_1,\cdots ,j_N}{U}_{d_1,j_1}^{(1)}\cdots U_{d_N,j_N}^{(N)}.$$

$$A_{d_{p_1},\cdots ,d_{p_{n_1}},d_{q_1},\cdots ,d_{q_{n_2}}}$$

$$=$$

$$[\mathcal{A}{]}^{(p,q)}$$

$$[\mathcal{A}{]}^{(p,q)}={\mathbf{U}}_{od}[\mathcal{G}{]}^{(p,q)}{\mathbf{U}}_{en}^T,$$

$${\mathbf{U}}_{od}={\mathbf{U}}^{(p_1)}\otimes {\mathbf{U}}^{(p_2)}\otimes \cdots \otimes {\mathbf{U}}^{(p_{n_1})}\in {\mathbb{R}}^{M^t\times J_{p_1}{J}_{p_2}\cdots J_{p_{n_1}}},$$

Full text

1 Introduction

Deep neural networks have achieved a great success in solving many practical problems. Deep learning methods are based on multiple levels of representation in learning. Each level involves simple but non-linear units for learning. Many deep learning networks have been developed and applied in various applications successfully. For example, convolutional neural networks (CNNs) [17, 21, 29] have been well applied in computer vision problems, recurrent neural networks (RNNs) [9, 12, 25] are used in audio and natural language processing. For more detailed discussions, see [22] and its references.

In the recent years, more and more works focus on the theoretical explanations of neural networks. One important topic is the expressive power, i.e., comparing the expressive ability of different neural networks architectures. In the literature [7, 8, 15, 16, 24, 26, 27, 28, 30], researches have been done in the investigation of the depth efficiency of neural networks. It is natural to claim that a deep network can be more powerful in the expressiveness than a shallow network. Recently, Khrulkov et al. [19] applied a tensor train decomposition to exploit the expressive power of RNNs experimentally. In [4], Cohen et al. theoretically analyzed specific shallow convolutional network by using CP decomposition and specific deep convolutional network based on hierarchical tensor decomposition. The result of the paper is that the expressive power of such deep convolutional networks is significantly better than that of shallow networks. Cohen et al. in [5] generalized convolutional arithmetic circuits into convolutional rectifier networks to handle activation functions, like ReLU. They showed that the depth efficiency of convolutional rectifier networks is weaker than that of convolutional arithmetic circuits.

Although many attempts in theoretical analysis success, the understanding of expressiveness is still needed to be developed. The main contribution of this paper is that a new deep network based on Tucker tensor decomposition is proposed. We analyze the expressive power of the new network, and show that it is required to use an exponential number of nodes in a shallow network to represent a Tucker network. Moreover, we compare the performance of the proposed Tucker network, hierarchical tensor network and shallow network on two datasets (Mnist and CIFAR) and demonstrate that the proposed Tucker network outperforms the other two networks.

The rest of this paper is organized as follows. In Section 2, we briefly review tensor decompositions. We present the proposed Tucker network and show its expressive power in Section 3. In Section 4, experimental results are presented to demonstrate the performance of the Tucker network. Some concluding remarks are given in Section 5.

2 Tensor Decomposition

A $N$-dimensional tensor $\mathcal{A}$ is a multidimensional array, i.e., $\mathcal{A}\in\mathbb{R}^{M_{1}\times M_{2}\times\cdots\times M_{N}}$. Its $i$-th unfolding matrix is defined as $A_{(i)}\in\mathbb{R}^{M_{i}\times M_{i+1}\cdots M_{N}M_{1}\cdots M_{i-1}}$. Given an index subset $\mathbf{p}=\{p_{1},p_{2},\cdots,p_{n_{1}}\}$ and the corresponding compliment set $\mathbf{q}=\{q_{1},q_{2},\cdots,q_{n_{2}}\}$, $n_{1}+n_{2}=N$, the $(p,q)$-matricization of $\mathcal{A}$ is denoted as a matrix $[\mathcal{A}]^{(p,q)}\in\mathbb{R}^{M_{p_{1}}\cdots M_{p_{n_{1}}}\times M_{q_{1}}\cdots M_{q_{n_{2}}}}$, obtained by reshaping tensor $\mathcal{A}$ into matrix.

We also introduce two important operators in tensor analysis, tensor product and Kronecker product. Given tensors $\mathcal{A}$ and $\mathcal{B}$ of order $N_{1}$ and $N_{2}$ respectively, the tensor product is defined as $(\mathcal{A}\circ\mathcal{B})_{d_{1},\cdots,d_{N_{1}+N_{2}}}=\mathcal{A}_{d_{1},\cdots,d_{N_{1}}}\mathcal{B}_{d_{N_{1}+1},\cdots,d_{N_{1}+N_{2}}}$. Note that when $N_{1}=N_{2}=1$, the tensor product is the outer product of vectors. $\otimes$ denotes Kronecker product which is an operation on two matrices, i.e., for matrices $\mathbf{A}\in\mathbb{R}^{m_{1}\times n_{1}}$, $\mathbf{B}\in\mathbb{R}^{m_{2}\times n_{2}}$, $\mathbf{A}\otimes\mathbf{B}\in\mathbb{R}^{m_{1}m_{2}\times n_{1}n_{2}}$, defined by $(\mathbf{A}\otimes\mathbf{B})_{m_{2}(r-1)+v,n_{2}(s-1)+w}=a_{rs}b_{vw}$.

Moreover, we use $[k]$ to denote the set $\{1,\cdots,k\}$ for simplicity.

In the following, we review some well-known tensor decomposition methods and related convolutional networks.

CP decomposition: [3, 14] Given a tensor $\mathcal{A}\in\mathbb{R}^{M_{1}\times M_{2}\times\cdots\times M_{N}}$, the CANDECOMP/PARAFAC decomposition (CP) is defined as follows:

[TABLE]

where $\mathbf{\lambda}^{y}\in\mathbb{R}^{Z}$, $\mathbf{a}^{z,i}\in\mathbb{R}^{M_{i}}$. The minimal value of $Z$ such that CP decomposition exists is called the CP rank of $\mathcal{A}$ denoted as $rank_{CP}(\mathcal{A})=Z$.

Tucker decomposition:[6, 31] Given a tensor $\mathcal{A}\in\mathbb{R}^{M_{1}\times M_{2}\times\cdots\times M_{N}}$, the Tucker decomposition is defined as follows:

[TABLE]

which can be written as,

[TABLE]

where $\mathcal{G}=(g_{j_{1},j_{2},\cdots,j_{N}})\in\mathbb{R}^{J_{1}\times J_{2}\times\cdots\times J_{N}}$, $\mathbf{u}^{(1)}_{j_{1}}\in\mathbb{R}^{M}$ , $\cdots$, $\mathbf{u}^{(N)}_{j_{N}}\in\mathbb{R}^{M}$, $j_{1}\in[J_{1}]$, $\cdots$, $j_{N}\in[J_{N}]$. The minimal value of $(J_{1},J_{2},\cdots,J_{N})$ such that (2) holds is called Tucker rank of $\mathcal{A}$, denoted as $rank_{Tucker}(\mathcal{A})=(J_{1},J_{2},\cdots,J_{N})$. If $rank_{Tucker}(\mathcal{A})=(J,\cdots J)$, we simplicity denoted as $rank_{Tucker}(\mathcal{A})=J$.

HT decomposition: The Hierarchical Tucker (HT) Tensor format is a multilevel variant of a tensor decomposition format. The definition requires the introduction of a tree. For detailed discussion, see [10, 11, 13]. Given a tensor $\mathcal{A}\in\mathbb{R}^{M_{1}\times M_{2}\times\cdots\times M_{N}}$, $N=2^{L}$. The $2^{L}$ hierarchical tensor decomposition has the following form:

[TABLE]

where $\{\mathbf{u}^{0,t_{1},\alpha}\}_{t_{1}\in[N],\alpha\in[r_{0}]}$ are the generated vectors of tensor $\mathcal{A}$. $r_{l}$ refer to level-$l$ rank. We denote $rank_{HT}(\mathcal{A})=(r_{0},r_{1},\cdots,r_{l-1})$. If all the ranks are equal to $r$, $rank_{HT}(\mathcal{A})=r$ for simple.

2.1 Convolutional Networks

Given a dataset of pairs $\{(X^{(b)},y^{(b)})\}^{D}_{b=1}$, each object $X^{(b)}$ is represented as a set of vectors $X^{(b)}=(\mathbf{x}_{1},\mathbf{x}_{2},\cdots,\mathbf{x}_{N})$ with ${\bf x}_{i}\in\mathbb{R}^{s}$. By applying parameter dependent functions $\{f_{\theta_{d}}:\mathbb{R}^{s}\rightarrow\mathbb{R}\}^{M}_{d=1}$, we construct a representation map $f_{\theta}:\mathbb{R}^{s}\rightarrow\mathbb{R}^{M}$. Object $X=(\mathbf{x}_{1},\mathbf{x}_{2},\cdots,\mathbf{x}_{N})$ with $\mathbf{x}_{i}\in\mathbb{R}^{s}$ is classified into one of categories $\mathfrak{Y}=\{1,2,\cdots,Y\}$. Classification is carried out through the maximization of the following score function:

[TABLE]

where $\mathcal{A}^{y}\in\mathbb{R}^{M\times M\times\cdots\times M}$ is a trainable coefficient tensor.

The representation functions $\{f_{\theta_{d}}:\mathbb{R}^{s}\rightarrow\mathbb{R}\}^{M}_{d=1}$ have many choices. For example, neurons-type functions $f_{\theta_{d}}(\mathbf{x})=\sigma(\mathbf{x}^{T}\mathbf{w}_{d}+\mathbf{b}_{d})$ for parameters $\theta_{d}=(\mathbf{w}_{d},\mathbf{b}_{d})$ and point-wise non-linear activation $\sigma(\cdot)$. We list some commonly used activation functions here, for example hard threshold: $\sigma(z)=1$ for $z>0$, otherwise 0; the rectified linear unit (ReLU) $\sigma(z)=\max\{z,0\}$; and sigmoid $\sigma(z)=\frac{1}{1+e^{-z}}$.

The main task is to estimate the parameters $\theta_{1},\theta_{2},\cdots,\theta_{M}$ and the coefficient tensors $\mathcal{A}^{1},\cdots,\mathcal{A}^{Y}$. The computational challenge is that the coefficient tensor has an exponential number of entries. We can utilize tensor decompositions to address this issue.

If the coefficient tensor is in CP decomposition, the network corresponding to CP decomposition is called shallow network(or CP Network), see Figure 1. We obtain its score function:

[TABLE]

Note that the same vectors $\mathbf{a}^{z,i}$ are shared across all classes $y$. If set $Z=M^{N}$, the model is universal, i.e., any tensors $\mathcal{A}^{1},\cdots,\mathcal{A}_{Y}$ can be represented.

If the coefficient tensors are in HT format like (8), the network refer to HT network. An example of HT network with $L=2$ is showed in Figure 2. Cohen et al. [4] analyzed the expressive power of HT network and proved that a shallow network with exponentially large width is required to emulate a HT network.

3 Tucker Network

In this section, we propose a Tucker network. If the coefficient tensors in (4) are in Tucker format (2), we refer it as Tucker network, i.e.,

[TABLE]

Suppose $J_{k}=\max_{y}rank(\mathbf{A}^{y}_{(k)})$ for same vectors $\mathbf{u}^{(k)}_{j_{k}}$($k\in[N]$) in (6). Here $\mathbf{A}^{y}_{(k)}$ be the $k$-th unfolding of tensor $\mathcal{A}^{y}$. If set $J_{1}=J_{2}=\cdots=J_{N}=J$, the number of parameter is: $YJ^{N}+MNJ$. If set $J_{1}=J_{2}=\cdots=J_{N}=M$, the model is universe, any tensor can be represented by Tucker format, number of $YM^{N}+M^{2}N$ parameters are needed. Note that the score function for Tucker network:

[TABLE]

The Tucker network architecture is given in Figure 3. The outputs from convolution layer are

[TABLE]

where $U^{(i)}=(u^{(i)}_{d,j_{i}})_{1\leq d\leq M;1\leq j_{i}\leq J_{i}}$, $i\in[N]$. The last output, i.e., score value is given as follows:

[TABLE]

where $\langle\cdot\rangle$ is tensor scalar product, i.e., the sum of entry-wise product of two tensors. Because $\mathcal{G}^{y}$ is a $N$ order tensor of smaller dimension $J_{1}\times\cdots\times J_{N}$, it can be further decomposed with a deeper network. In this sense, Tucker network is also a kind of deep network.

The following theorem demonstrates the expressive power of Tucker network.

Theorem 1**.**

Let $\mathcal{A}^{y}$ be a tensor of order $N$ and dimension $M$ in each mode, generated by Tucker form in (6). Define $r=\max_{p,q}[\mathcal{G}]^{(p,q)}$ for all possible subsets $\mathbf{p},\mathbf{q}$, consider the space of all possible configurations for parameters. In the space, $\mathcal{A}^{y}$ will have CP rank of at least $r$ almost everywhere, i.e.,the Lebesgue measure of the space whose CP rank is less than $r$ is zero.

The proof can be found in the supplementary section. We remark that if $J_{1}=J_{2}=\cdots=J_{N}=J$, when $N$ is even, the Lebesgue measure of the Tucker format space whose CP rank is less than $J^{\frac{N}{2}}$ is zero; when $N$ is odd, the Lebesgue measure of the Tucker format space whose CP rank is less than $J^{\frac{N-1}{2}}$ is also zero.

3.1 Connection with HT Network

In this subsection, to compare the expressive power of HT and Tucker network, we discuss the relationship between Tucker format and hierarchical Tucker tensor format firstly. Here we only consider $N=2^{L}$ hierarchical tensor format, its corresponding HT network (8) has been well discussed in [4].

We start it from $L=2$ hierarchical Tucker tensor, its HT network architecture is shown in Figure 2 . Given a $2^{2}$ order tensor, its hierarchical tensor format can always be written as

[TABLE]

$\mathbf{u}^{0,i,\alpha},i\in[4]$ are vectors size of $M$. Here we suppose that $M\geq r_{0}$. Denote $\mathbf{A}=[\mathcal{A}]^{(12,34)}$, we have $\mathbf{A}=\Phi^{1,1}\Sigma_{1}(\Phi^{1,2})^{T}$, where

[TABLE]

[TABLE]

[TABLE]

$vec(\cdot)$ is a linear transformation that converts a matrix into a column vector. $diag(\cdot)$ is diagonal operator that transform a vector into a diagonal matrix. Similarly, we have,

[TABLE]

where \mathbf{U}^{0,i}=\left(\begin{array}[]{ccc}\mathbf{u}^{0,i,1}&\cdots&\mathbf{u}^{0,i,r_{0}}\\ \end{array}\right), $i=[4]$, and $\Sigma^{0,j,\gamma_{1}}=diag(a^{1,j,\gamma_{1}}_{1},\cdots,a^{1,j,\gamma_{1}}_{r_{0}})$, $j=[2],\gamma_{1}\in[r_{1}]$.

From the property of Kronecker product: $\mathbf{Y}=\mathbf{A}\mathbf{X}\mathbf{B}\Leftrightarrow vec(\mathbf{Y})=(\mathbf{B}^{T}\otimes\mathbf{A})vec(\mathbf{X})$, we deduce that,

[TABLE]

Therefore,

[TABLE]

with \mathbf{D}_{1}=\left(\begin{array}[]{ccc}vec(\Sigma^{0,1,1})&\cdots&vec(\Sigma^{0,1,r_{1}})\\ \end{array}\right), \mathbf{D}_{2}=\left(\begin{array}[]{ccc}vec(\Sigma^{0,2,1})&\cdots&vec(\Sigma^{0,2,r_{1}})\\ \end{array}\right). We can get that

[TABLE]

Therefore,

[TABLE]

with $\mathcal{G}^{(12,34)}=D_{1}\Sigma_{1}D^{T}_{2}$. It implies that a $2^{2}$ hierarchical tensor format can be written as a $2^{2}$ order Tucker tensor. Worth to say, from (7), the rank of $\mathbf{A}$ is less than that of its factor matrices. Because of the structure of $\mathbf{D}_{1}$, $\mathbf{D}_{2}$, we get that $rank(\mathbf{D}_{1})\leq\min\{r_{0},r_{1}\}$ and also $rank(\mathbf{D}_{2})\leq\min\{r_{0},r_{1}\}$. From the rank property, $rank(\mathbf{A})\leq\min\{r_{0},r_{1}\}$.

When the hierarchical tensor has $L$ layers, we can similarly deduced the following results.

Theorem 2**.**

Any $2^{L}$ hierarchical tensor can be represented as a $2^{L}$ order Tucker tensor and vice versa.

Theorem 3**.**

For any tensor $\mathcal{A}$, if $rank_{HT}\leq r$, then $rank_{Tucker}\leq r$.

The detailed proofs of Theorem 2 and Theorem 3 can be found in Appendix.

According to Theorem 3, given a hierarchical Tucker network of width $r$, we know that the width of Tucker network is not possible larger than $r$.

4 Experimental Results

We designed experiments to compare the performance of three networks: Tucker network, HT network and shallow network. The results illustrate the usefulness of Tucker network. We implement shallow network, Tucker network and HT network with TensorFlow[1] back-end, and test three networks on two different data sets: Mnist [23] and CIFAR-10 [20]. All three networks are trained by using the back-propagation algorithm. In all three networks, we choose ReLU as the activation function in the representation layer and apply batch normalization [18] between convolution layer and pooling layer to eliminate numerical overflow and underflow.

We choose Neurons-type $f_{\theta}(\mathbf{x})=\sigma(\mathbf{x}^{T}\mathbf{w}+\mathbf{b})$ with ReLU nonlinear activation $\sigma$ as representation map $\{f_{\theta_{d}}$: $\mathbb{R}^{s}\rightarrow\mathbb{R}\}_{d=1}^{M}$. Actually the representation mapping now is acted as a convolution layer in general CNNs. Each image patch is transformed through a representation function with parameter sharing across all the image patches. Convolution layer $\mathbf{v}^{(i)}=U^{(i)T}f_{\theta}(\mathbf{x}_{i}),i\in[N]$ in Figure 3 actually can been seen as a locally connected layer in CNN. It is a specific convolution layer without parameter sharing, which means that the parameters of filter would differ when sliding across different spatial positions. In the hidden layer, without overlapping, a 3D convolution operator size of $1\times 1\times j_{i}$ is applied. Following is a product pooling layer to realize the outer product computation $\prod^{N}_{i=1}\mathbf{v}^{(i)}_{j_{i}}$. It can be explained as a pooling layer with local connectivity property, which only connects a neuron with partial neurons in the previous layer. The output of neuron is the multiplication of entries in the neurons connected to it. The fully-connected layer simply apply the linear mapping on the output of pooling layer. The output of Tucker network would be a vector $Y$ corresponding to class scores.

4.1 Mnist

The MNIST database of handwritten digits has a training set of 60000 examples, and a test set of 10000 examples with 10 categories from 0 to 9. Each image is of $28\times 28$ pixels. In the experiment, we select the gradient descent optimizer for back-propagation with batch size 200, and use a exponential decay learning rate with 0.2 initial learning rate, 6000 decay step and 0.1 decay rate. Figure 4 shows the training and test accuracy of three networks with 3834 number of parameters that have been learned. The parameters contains four parameters in batch normalization (mean, std, alpha, beta). We list filter size, strides size and rank as well in Table 1. It is obvious that Tucker network outperforms shallow network and HT network. Moreover, we test the sensitivity of Tucker network with the change of rank, and compare the performance with the other two networks with the same number of parameters. Figure 5 illustrates the sensitivity performance, each value records the highest accuracy in training or test data. Tucker network can achieve the highest accuracy at most times.

4.2 CIFAR-10

CIFAR-10 data [20] is a more complicated data set consisting of 60000 color images size of $32\times 32$ with 10 classes. Here, we use the gradient descent optimizer with 0.05 learning rate and 200 batch size to train. In Figure 6 we report the training and test accuracy with 23790 trained parameters. Table 2 shows the parameter details of sensitivity test, whose results are displayed in Figure 7 . From Figure 6 and Figure 7 , Tucker network still has more excellent performance when fitting a more complicated data set.

5 Conclusion

In this paper, we presented a Tucker network and prove the expressive power theorem. We stated that a shallow network of exponentially large width is required to mimic Tucker network. A connection between Tucker network and HT network is discussed. The experiments on Mnist and CIFAR-10 data show the usefulness of our proposed Tucker network.

Appendix A Appendix A. Proofs

A.1 Proof of Theorem 1

In section 3, we presented Tucker network and showed its expressive power. To prove Theorem 1, we firstly state and prove three lemmas which will be needed for the proofs.

Lemma 1**.**

For any $(p,q)$ matricization of a tensor $\mathcal{A}$ whose CP rank is $Z$,

[TABLE]

Proof.

[TABLE]

∎

In Lemma 1, we give the lower bound of the CP-rank. If the matricization of a tensor $\mathcal{A}$ has matrix rank $Z$, then using the above lemma, we get that the CP rank of $\mathcal{A}$ is larger than $Z$.

For a $N$ order tensor $\mathcal{A}$ who is in Tucker format, its matricization has the following form.

Lemma 2**.**

Given a $N$ order tensor $\mathcal{A}\in\mathcal{R}^{M\times M\times\cdots\times M}$ whose Tucker format is $\mathcal{A}=\mathcal{G}\times_{1}\mathbf{U}^{(1)}\times\cdots\times_{N}\mathbf{U}^{(N)}$, index subsets $\mathbf{p}=\{p_{1},\cdots,p_{n_{1}}\}$ and $\mathbf{q}=\{q_{1},\cdots,q_{n_{2}}\}$, then

[TABLE]

Proof.

[TABLE]

Therefore,

[TABLE]

where $H_{d_{p_{1}},\cdots,d_{p_{n_{1}}},j_{q_{1}},\cdots,j_{q_{n_{2}}}}=\sum_{j_{p_{1}},\cdots,j_{p_{n_{1}}}}g_{j_{p_{1}},\cdots,j_{p_{n_{1}}},j_{q_{1}},\cdots,j_{q_{n_{2}}}}U^{(p_{1})}_{d_{p_{1}},j_{p_{1}}}\cdots U^{(p_{n_{1}})}_{d_{p_{n_{1}}},j_{p_{n_{1}}}}$. Then,

[TABLE]

∎

For simplicity, we denote

[TABLE]

where

[TABLE]

[TABLE]

[TABLE]

We get

[TABLE]

[TABLE]

Lemma 3**.**

If each factor matrix of tensor $\mathcal{A}$ has full column rank, i.e., $\mathbf{U}^{(1)},\cdots,\mathbf{U}^{(N)}$ has full column rank, then $rank([\mathcal{A}]^{(p,q)})=rank([\mathcal{G}]^{(p,q)})$.

Proof.

$rank([\mathcal{A}]^{(p,q)})=rank(\mathbf{U}_{od}[\mathcal{G}]^{(p,q)}\mathbf{U}^{T}_{en})=rank([\mathcal{G}]^{(p,q)}\mathbf{U}^{T}_{en})=rank([\mathcal{G}]^{(p,q)})$. ∎

Proof of Theorem 1

Proof.

According to Lemma 1, it suffices to prove that the rank of $[\mathcal{A}^{y}]^{(p,q)}$ is at least $r$ almost everywhere. From Lemma 3, equivalently, we prove the rank of $[\mathcal{G}]^{(p,q)}$ is at least $r$ almost everywhere.

For any $x\in\mathbb{R}^{J_{p_{1}}J_{p_{2}}\cdots J_{p_{n_{1}}}J_{q_{1}}J_{q_{2}}\cdots J_{q_{n_{2}}}}$, and all possible subsets $\mathbf{p}=\{p_{1},p_{2},\cdots,p_{n_{1}}\}$ and the corresponding compliment set $\mathbf{q}=\{q_{1},q_{2},\cdots,q_{n_{2}}\}$, $n_{1}+n_{2}=N$. We let $[\mathcal{G}]^{(p,q)}(x)\in\mathbb{R}^{J_{p_{1}}J_{p_{2}}\cdots J_{p_{n_{1}}}\times J_{q_{1}}J_{q_{2}}\cdots J_{q_{n_{2}}}}$, which simply holds the elements of $x$. Because $r=\max_{p,q}[\mathcal{G}]^{(p,q)}$ for all possible subsets $\mathbf{p},\mathbf{q}$. For all $x$, we have $rank([\mathcal{G}]^{(p,q)}(x))\leq r$. In the following, we will prove that the Lebesgue measure of the space that $rank([\mathcal{G}]^{(p,q)}(x))<r$ is zero.

Let $([\mathcal{G}]^{(p,q)}(x))_{r}$ be the top-left $r\times r$ sub matrix of $[\mathcal{G}]^{(p,q)}(x)$ and $det(([\mathcal{G}]^{(p,q)}(x))_{r})$ is the determinant, as we know that $det(([\mathcal{G}]^{(p,q)}(x))_{r})$ is a polynomial in the entries of $x$, according to theorem in[2], it either vanishes on a set of zero measure or it is the zero polynomial. It implies that the Lebesgue measure of the space whose $det(([\mathcal{G}]^{(p,q)}(x))_{r})=0$ is zero, i.e., the Lebesgue measure of the space whose rank less than $r$ is zero. The result thus follows. ∎

A.2 Proof of Theorem 2

In this subsection, we will prove Theorem 2, the connection of Tucker tensor format and $2^{L}$ hierarchical Tucker tensor format. The expressive power of hierarchical Tucker tensor network has been well discussed in [4].

In Section 2, we defined $(p,q)$-matricization which is a kind of general matricization. In the following, we simply consider the proper order matricization of tensor, denoted as the matrix $[\mathcal{A}]_{p}$ here, for example, for $N=2t$, $[\mathcal{A}]_{p}\in\mathbb{R}^{M_{1}\cdots M_{t}\times M_{t+1}\cdots M_{2t}}$; for $N=2t+1$, $[\mathcal{A}]_{p}\in\mathbb{R}^{M_{1}\cdots M_{t}\times M_{t+1}\cdots M_{2t+1}}$.

The $2^{L}$ hierarchical tensor decomposition format is given as follows:

[TABLE]

It is easy to check that $\{\phi^{l,t_{l},\gamma_{l}}\}^{L-1}_{l=1}$ are tensors. The vectorization of these tensors $\{\phi^{l,t_{l},\gamma_{l}}\}^{L-1}_{l=1}$, are denoted as $\{\phi^{l,t_{l},\gamma_{l}}_{c}\}^{L-1}_{l=1}$ for simplicity. The matrix $[\phi^{l,t_{l},\gamma_{l}}]_{p}$ of tensor $\phi^{l,t_{l},\gamma_{l}}$ has $M_{1}M_{2}\cdots M_{2^{l-1}}$ rows and $M_{2^{l-1}+1}M_{2^{l-1}+2}\cdots M_{2^{l}}$ columns. Note that each dimension of $\mathcal{A}$ is equal to $M$, the matrix $[\phi^{l,t_{l},\gamma_{l}}]_{p}$ contains $M^{2^{l-1}}$ rows and $M^{2^{l-1}}$ columns. At the $(L-1)$-th layer, define,

[TABLE]

At the $l$-th layer, $l=2,\cdots,L-2$,

[TABLE]

At the $1$-st layer,

[TABLE]

Proof of Theorem 2

Proof.

From (8), for the case $l=1$, we have

[TABLE]

Denote $\mathbf{U}^{0,2t_{1}-1}\in\mathbb{R}^{M\times r_{0}}$ the matrix with columns $\{\mathbf{u}^{0,2t_{1}-1,\alpha}\}^{r_{0}}_{\alpha=1}$, $\mathbf{U}^{0,2t_{1}}\in\mathbb{R}^{M\times r_{0}}$ the matrix with columns $\{\mathbf{u}^{0,2t_{1},\alpha}\}^{r_{0}}_{\alpha=1}$, and $\mathbf{D}^{1,t_{1},\gamma_{1}}\in\mathbb{R}^{r_{0}\times r_{0}}$ the diagonal matrix with $a^{1,t_{1},\gamma_{1}}_{\alpha}$ on its diagonal. We rewrite

[TABLE]

therefore,

[TABLE]

[TABLE]

Here, for simplicity, we denote $(vec(\mathbf{D}^{l,t_{l},1}),\cdots,vec(\mathbf{D}^{l,t_{l},r_{l}}))$ as $\mathbf{P}^{l,t_{l}}$.

For the case $l=2$,

[TABLE]

then,

[TABLE]

[TABLE]

therefore,

[TABLE]

[TABLE]

where $\tilde{\mathbf{P}}^{2,1,\gamma_{2}}=\mathbf{P}^{1,1}\mathbf{D}^{2,1,\gamma_{2}}(\mathbf{P}^{1,2})^{T},\cdots,\tilde{\mathbf{P}}^{2,2^{L-2},\gamma_{2}}=\mathbf{P}^{1,2^{L-1}-1}\mathbf{D}^{2,2^{L-2},\gamma_{2}}(\mathbf{P}^{1,2^{L-1}})^{T}$.

Assume for the case $l=l$, $\Phi^{l,t}$ always be written as

[TABLE]

[TABLE]

For the case $l=l+1$,

[TABLE]

[TABLE]

then

[TABLE]

[TABLE]

where $(vec(\tilde{\mathbf{P}}^{l,t_{l},1}),\cdots,vec(\tilde{\mathbf{P}}^{l,t_{l},r_{l}}))$ is denoted as $\tilde{\mathbf{P}}^{l,t_{l}}$.

From the above, we can deduce that

[TABLE]

i.e.,

[TABLE]

where $[\mathcal{G}]_{p}=\tilde{\mathbf{P}}^{L-1,1}\mathbf{D}^{L,1}(\tilde{\mathbf{P}}^{L-1,2})^{T}$. ∎

Proof of Theorem 3

From the proof of Theorem 2, we know that, $rank_{Tucker}\mathcal{A}\leq r_{0}$, here $r_{0}$ is the rank of $1$-st layer of hierarchical Tucker tensor. Because of the assumption $rank_{HT}\mathcal{A}\leq r$, we get that $r_{0}\leq r$. $rank_{Tucker}\mathcal{A}\leq r_{0}\leq r$ established.