Classification and Representation via Separable Subspaces: Performance Limits and Algorithms

TL;DR

This paper analyzes the classification limits of Kronecker-structured models, derives exact performance metrics, and introduces a fast algorithm for learning discriminative dictionaries that improve classification of multidimensional signals.

Contribution

It provides a theoretical analysis of classification performance in K-S models and proposes a novel, efficient algorithm for dictionary learning tailored to these models.

Findings

Derived exact diversity order as a function of signal and subspace dimensions.

Established the maximum classification capacity as signal dimension increases.

Demonstrated the effectiveness of K-SLD2 in synthetic and real-world datasets.

Abstract

We study the classification performance of Kronecker-structured models in two asymptotic regimes and developed an algorithm for separable, fast and compact K-S dictionary learning for better classification and representation of multidimensional signals by exploiting the structure in the signal. First, we study the classification performance in terms of diversity order and pairwise geometry of the subspaces. We derive an exact expression for the diversity order as a function of the signal and subspace dimensions of a K-S model. Next, we study the classification capacity, the maximum rate at which the number of classes can grow as the signal dimension goes to infinity. Then we describe a fast algorithm for Kronecker-Structured Learning of Discriminative Dictionaries (K-SLD2). Finally, we evaluate the empirical classification performance of K-S models for the synthetic data, showing that they agree with the diversity order analysis. We also evaluate the performance of K-SLD2 on synthetic and real-world datasets showing that the K-SLD2 balances compact signal representation and good classification performance.

Tables

Table 1. TABLE I : Comparison between different approaches for extended YaleB face recognition dataset.

	SSL	DLSI	FDDL	LRSDL	SVM	K-SLD2
Test sample classification accuracy (%)	79.36	85.62	88.43	88.14	80.43	88.86
Number of parameters for representation	102400	102400	102400	$\sim$102400	$\sim$102400	9600
Average training time (sec)	0.034	1.2697	9.3254	50.5129	2.234	0.111035
Normalized reconstruction error	0.290	0.363	0.346	0.376	—	0.178

Table 2. TABLE II : Comparison between different approaches on EEG signal dataset

	SSL	DLSI	FDDL	LRSDL	SVM	K-SLD2
Test sample classification accuracy (%)	64	66.25	67.5	65	64.9	68.25
Number of parameters for representation	163840	163840	163840	$\sim$ 163840	$\sim$ 163840	2176
Average training time (sec)	0.012	0.9546	3.4301	128.6921	1.12	0.04
Normalized reconstruction error	0.290	0.363	0.346	0.50	—	0.178

Equations

$$\arg \underset{\{D_i,x_i\}}{\min }\sum _i||y_i-D_i{x}_i|{|}_F^2+\lambda ||x_i|{|}_1$$

$$\mathbf{Y}={\mathbf{A}}_l\mathbf{X}{\mathbf{B}}_l^T+\mathbf{Z},$$

$$\mathbf{y}=({\mathbf{B}}_l\otimes {\mathbf{A}}_l)\mathbf{x}+\mathbf{z},$$

$$p_l(\mathbf{y})=\mathcal{N}(0,({\mathbf{B}}_l\otimes {\mathbf{A}}_l)({\mathbf{B}}_{\mathbf{l}}\otimes {\mathbf{A}}_{\mathbf{l}}{)}^T+{\sigma }^2\cdot \mathbf{I}).$$

$$\mathcal{A}(m_1,m_2,n_1,n_2)={\mathbb{R}}^{m_1\times n_1}\times {\mathbb{R}}^{m_2\times n_2},$$

$$P_e(\mathbf{a})=\frac{1}{L}\sum _{l=1}^L\mathrm{Pr}(\hat{l}\ne l|\mathbf{y}\sim p(\mathbf{y}|a_l),$$

$$p(\mathbf{a})=\prod _{p=1}^{m_1}\prod _{q=1}^{n_1}\prod _{r=1}^{m_2}\prod _{s=1}^{n_2}\mathcal{N}(a_{pq};0,1/n_1)\cdot \mathcal{N}(b_{rs};0,1/n_2)$$

$$d(\mathbf{a})=\underset{{\sigma }^2\to 0}{\lim }-\frac{\log P_e(\mathbf{a})}{\frac{1}{2}\log (1/{\sigma }^2)}.$$

$$\begin{gathered} m_1(m)=\lfloor {\kappa }_{1}m\rfloor ,m_2(m)=\lfloor {\kappa }_{2}m\rfloor , \\ {n}_1(m)=\lfloor {\upsilon }_{1}m\rfloor ,n_2(m)=\lfloor {\upsilon }_{2}m\rfloor \end{gathered}$$

$$L(m)=\lfloor 2^{\rho m_1(m)m_2(m)}\rfloor ,$$

$$C\le \underset{m\to \infty }{\lim }\frac{I(\mathbf{y};\mathbf{A},\mathbf{B})}{m_1(m)m_2(m)}$$

$$b=\frac{1}{2}\ln \left(\frac{|\frac{{\Sigma }_1+{\Sigma }_2}{2}|}{|{\Sigma }_1{|}^{\frac{1}{2}}|{\Sigma }_2{|}^{\frac{1}{2}}}\right)+\frac{1}{8}({\mu }_1-{\mu }_2){\left[\frac{{\Sigma }_1+{\Sigma }_2}{2}\right]}^{-1}({\mu }_1-{\mu }_2)$$

$$P_e({\mu }_1,{\Sigma }_1,{\mu }_2,{\Sigma }_2)\le \frac{1}{2}\exp (-b).$$

$$\cos ({\theta }_t)$$

$$\text{subject to }{u}_t^T{u}_t=1,v_t^T{v}_t=1,$$

$$u_t^T{u}_i=0,v_t^T{v}_t=0,(i<t)$$

$${\mathbf{A}}_1^T{\mathbf{A}}_2={\mathbf{U}}_A\cos ({\Theta }_A){\mathbf{V}}_A^T,$$

$${\mathbf{D}}_1^T{\mathbf{D}}_2$$

$$=({\mathbf{B}}_1^T{\mathbf{B}}_2)\otimes ({\mathbf{A}}_1^T{\mathbf{A}}_2)$$

$$=({\mathbf{U}}_B\cos ({\Theta }_B){\mathbf{V}}_B^T)\otimes ({\mathbf{U}}_A\cos ({\Theta }_A){\mathbf{V}}_A^T)$$

$$=({\mathbf{U}}_B\otimes {\mathbf{U}}_A)(\cos ({\Theta }_B)\otimes \cos ({\Theta }_A))({\mathbf{V}}_B\otimes {\mathbf{V}}_A{)}^T$$

$$=\mathbf{U}\cos (\Theta ){\mathbf{V}}^T,$$

$$r^{\ast }=\underset{i,j}{\min }r(\left[\begin{array}{cc}{\mathbf{B}}_i\otimes {\mathbf{A}}_i& {\mathbf{B}}_j\otimes {\mathbf{A}}_j\end{array}\right]),$$

$${\Sigma }_i={\mathbf{D}}_i{\mathbf{D}}_i^T+{\sigma }^2\mathbf{I},{\Sigma }_j={\mathbf{D}}_j{\mathbf{D}}_j^T+{\sigma }^2\mathbf{I},$$

$$P_e({\mathbf{D}}_i,{\mathbf{D}}_j)\le \frac{1}{2}{\left(\frac{|\frac{{\mathbf{D}}_i{\mathbf{D}}_i^T+{\mathbf{D}}_j{\mathbf{D}}_j^T+2{\sigma }^2\mathbf{I}}{2}|}{|{\mathbf{D}}_i{\mathbf{D}}_i^T+{\sigma }^2\mathbf{I}{|}^{\frac{1}{2}}|{\mathbf{D}}_j{\mathbf{D}}_j^T+{\sigma }^2\mathbf{I}{|}^{\frac{1}{2}}}\right)}^{-\frac{1}{2}}$$

$${\mathbf{D}}_i{\mathbf{D}}_i^T$$

$${\mathbf{D}}_j{\mathbf{D}}_j^T$$

$${\mathbf{D}}_i{\mathbf{D}}_i^T+{\mathbf{D}}_j{\mathbf{D}}_j^T=\left[\begin{array}{cc}{\mathbf{B}}_i\otimes {\mathbf{A}}_i& {\mathbf{B}}_j\otimes {\mathbf{A}}_j\end{array}\right]\cdot \left[\begin{array}{c}{\mathbf{B}}_i^T\otimes {\mathbf{A}}_i^T\\ {\mathbf{B}}_j^T\otimes {\mathbf{A}}_j^T\end{array}\right]$$

$$r({\mathbf{D}}_i{\mathbf{D}}_i^T+{\mathbf{D}}_j{\mathbf{D}}_j^T)=r(\left[\begin{array}{cc}{\mathbf{B}}_i\otimes {\mathbf{A}}_i& {\mathbf{B}}_j\otimes {\mathbf{A}}_j\end{array}\right])=r_{ij}^{\ast }$$

$$P_e({\mathbf{D}}_i,{\mathbf{D}}_j)$$

$$=\frac{1}{2}{\left(\frac{1}{{\sigma }^2}\right)}^{-\frac{r_{ij}^{\ast }-n_1{n}_2}{2}}\cdot {\left(\frac{{\prod }_{l=1}^{r_{ij}^{\ast }}({\lambda }_{ijl}+{\sigma }^2)}{\sqrt{{\prod }_{l=1}^{n_1{n}_2}({\lambda }_{il}+{\sigma }^2)\cdot {\prod }_{l=1}^{n_1{n}_2}({\lambda }_{jl}+{\sigma }^2)}}\right)}^{-\frac{1}{2}}$$

$${\mathbf{D}}_i{\mathbf{D}}_i^T+{\mathbf{D}}_j{\mathbf{D}}_j^T\ge {\mathbf{D}}_i{\mathbf{D}}_i^T,{\mathbf{D}}_j{\mathbf{D}}_j^T$$

$$\prod _{l=1}^{n_1{n}_2}2({\lambda }_{ijl}+{\sigma }^2)\ge \sqrt{\prod _{l=1}^{n_1{n}_2}({\lambda }_{il}+{\sigma }^2)\cdot \prod _{l=1}^{n_1{n}_2}({\lambda }_{jl}+{\sigma }^2)}$$

$$P_e({\mathbf{D}}_i,{\mathbf{D}}_j)$$

Full text

I Introduction

The classification of high-dimensional signals arises in a variety of image processing settiings: object and digit recognition [1, 2], speaker identification [3, 4], tumor classification [5, 6], and more. A standard technique is to find a low-dimensional representation of the signal, such as a subspace or union of subspaces on which the signal approximately lies. However, for many signals, such as dynamic scene videos [7] or tomographic images [8], the signal inherently is multi-dimensional, involving dimensions of space and/or time. To use standard techniques, one vectorizes the signal, which throws out the spatial structure of the data which could be leveraged to improve representation fidelity, reconstruction error, or classification performance.

In order to exploit multi-dimensional signal structure, researchers have proposed tensor-based dictionary learning techniques, in which the signal of interest is a matrix or a higher-order tensor and the dictionary defining the (union of) subspace model is a tensor. A simple tensor-based model is the Kronecker-structured (K-S) model, in which a two-dimensional signal is represented by a coefficient matrix and two matrix dictionaries that pre- and post-multiply the coefficient matrix, respectively. Vectorizing this model leads to a dictionary that is the Kronecker product of two smaller dictionaries; hence the K-S model is a specialization of subspace models. This model is applied to spatio-temporal data in [9], low-complexity methods for estimating K-S covariance matrices are developed in [10], and it is shown that the sample complexity of K-S models is smaller than standard union-of-subspace models in [11].

As standard union-of-subspace models have proven successful for classification tasks [12, 13, 14], a natural question is the classification performance of K-S subspace models. In this paper, we address this question from an information-theoretic perspective and developed an algorithm for learning discriminative K-S dictionaries. We consider a signal model in which each signal class is associated with a subspace whose basis is the Kronecker product of two smaller dictionaries; equivalently, we suppose that each signal class has a matrix normal distribution, where the row and column covariances are approximately low rank. Here the covariance of signal class follows a specific structure which is exactly the Kronecker product of two lower dimensional covariance matrices [15, 16, 17]. In this sense, signals are drawn from a matrix Gaussian mixture model (GMM), similar to [18], where each K-S subspace is associated with a mixture component.

To find the underlying low dimensional representation of signals, dictionary learning methods are widely used [19, 20, 21]. The underlying signal is compactly represented by a few large coefficients in an overcomplete dictionary. In a standard dictionary learning setting a 1-D signal $y_{i}$ is represented using a sparse coefficient vector $x_{i}$, where an overcomplete dictionary $D_{i}$ is learned by minimization problems similar to

[TABLE]

Where $||\cdot||_{F}$ denotes the Forbenius norm, $||\cdot||_{1}$ denotes the $l_{1}$-norm, and $\lambda$ denotes the strength of the sparsity prior. Well-established methods for dictionary learning in this framework include K-SVD [22] and the method of optimal directions [23]. These methods are targeted at dictionaries that faithfully represent the signal, and do not specifically consider classification.

Methods for incorporating discriminative ability into dictionary learning have been proposed, such as discriminative K-SVD [12] (D-KSVD) and label consistent (LC-KSVD) [24], which jointly learn a linear classifier and an overcomplete dictionary that is shared in common among the classes. Signals are then classified in the feature space induced by the dictionary. By contrast, [25, 13, 26, 27] propose methods for learning class-specific dictionaries, either by promoting incoherence among dictionaries or learning class-specific features. Signals are then classified by choosing the dictionary that minimizes the reconstruction error.

The above methods consider one-dimensional signals; multidimensional signals must first be vectorized, which may sacrifice structural information about the signal that could improve signal representation or classification. To preserve signal structure,[28] extends K-SVD to tensor dictionaries, and [29, 30, 31, 6] employ a variety of tensor decompositions to learn dictionaries tailored to multidimensional structure. These methods boast improved performance over traditional methods on a variety of signal processing tasks, including image reconstruction, image denoising and inpainting, video denoising, and speaker classification.

Similar to [32], we first study the classification performance limits of K-S models in terms of diversity order and classification capacity, characterizing the performance in the limit of high SNR and large signal dimension, respectively. Further, we derive a tight upper bound on the misclassification probability in terms of the pairwise geometry of individual row and column subspaces. Where row and column subspaces correspond to two matrix dictionaries that pre- and post-multiply the coefficient matrix, respectively. We use principal angles between the subspaces as a measure to describe the geometry of subspaces [33, 34].

Finally, to learn discriminative dictionaries, we propose a new method, termed Kronecker-Structured Learning of Discriminative Dictionaries (K-SLD2), that exploit multidimensional structure of the signal. K-SLD2 learns two subspace dictionaries per class: one to represent the columns of the signal, and one to represent the rows. Inspired by [26], we choose dictionaries that both represent each class individually and can be concatenated to form an overcomplete dictionary to represent signals generally. K-SLD2 is fast and learns compact data models with many fewer parameters than standard dictionary learning methods. We evaluate the performance of K-SLD2 on the Extended YaleB and UCI EEG database. The resulting dictionaries improve classification performance by up to 5% when training sets are small, improve reconstruction performance across the board, and result in dictionaries with no more than 5% of the storage requirements of existing subspace models.

In Section II, we describe the K-S classification model in detail. In Section III we derive the diversity order for K-S classification problems, showing the exponent of the probability of error as the SNR goes to infinity. This analysis depends on a novel expression, presented in Lemma 3, for the rank of sums of Kronecker products of tall matrices. In Section IV we provide high-SNR approximations to the classification capacity. In Section V, we propose a discriminative K-S dictionary learning algorithm which balances the learning of class-specific, Kronecker-structured subspaces against the learning of an general overcomplete dictionary that allows for the representation of general signals. In Section VI we show that the empirical classification performance of K-S models agrees with the diversity analysis and evaluate the performance of proposed discriminative algorithm on extended YaleB face recognition dataset and EEG signal dataset correlating the EEG signals with individual’s alcoholism.

II Problem Definition

II-A Kronecker-structured Signal Model

To formalize the classification problem, let the signal of interest $\mathbf{Y}\in\mathbb{R}^{m_{1}\times m_{2}}$ be a matrix whose entries are distributed according to one of $L$ class-conditional densities $p_{l}(\mathbf{Y})$. Each class-conditional density corresponds to a Kronecker-structured model described by the pair of matrices $\mathbf{A}_{l}\in\mathbb{R}^{m_{1}\times n_{1}}$ and $\mathbf{B}_{l}\in\mathbb{R}^{m_{2}\times n_{2}}$. The matrix $\mathbf{A}_{l}$ describes the subspace on which the columns of $\mathbf{Y}$ approximately lie, and $\mathbf{B}_{l}$ describes the subspace on which the rows of $\mathbf{Y}$ approximately lie. More precisely, if $\mathbf{Y}$ belongs to class $l$, it has the form

[TABLE]

where $\mathbf{Z}\in\mathbb{R}^{m_{1}\times m_{2}}$ has i.i.d. zero-mean Gaussian entries with variance $\sigma^{2}>0$, and $\mathbf{X}\in\mathbb{R}^{n_{1}\times n_{2}}$ has i.i.d. zero-mean Gaussian entries with unit variance. We can also express $\mathbf{Y}$ in vectorized form:

[TABLE]

for coefficient vector $\mathbf{x}=\mathrm{vec}(\mathbf{X})\,\in\mathbb{R}^{N}$, and noise vector $\mathbf{z}\in\mathbb{R}^{M}$, where $N=n_{1}n_{2}$, $M=m_{1}m_{2}$, and where $\otimes$ is the usual Kronecker product. Then, the class-conditional density of $\mathbf{y}$ is

[TABLE]

In other words, the vectorized signal $\mathbf{y}$ lies near a subspace with a Kronecker structure that encodes the row and column subspaces of $\mathbf{Y}$.

In the sequel, we will characterize the performance limits over ensembles of classification problems of this form. To this end, we parameterize the set of class-conditional densities via

[TABLE]

which contains the set of matrices indicating the row and column subspaces given signal and subspace dimensions $m_{1},m_{2},n_{1},n_{2}$. We can represent an $L$-ary classification problem by a tuple $\mathbf{a}=(a_{1},\cdots,a_{L})\in\mathcal{A}^{L}(m_{1},m_{2},n_{1},n_{2})$, where each $a_{l}\in\mathcal{A}(m_{1},m_{2},n_{1},n_{2})$ is the pair of matrices $a_{i}=(\mathbf{A}_{l},\mathbf{B}_{l})$. Let $p(\mathbf{y}|a_{l})=p(\mathbf{y}|\mathbf{A}_{l},\mathbf{B}_{l})=p_{l}(\mathbf{y})$, for $1\leq l\leq L$, denote the class conditional densities parametrized by $\mathbf{a}\in\mathcal{A}(m_{1},m_{2},n_{1},n_{2})$. For a classification problem defined by $\mathbf{a}$, we can define the average misclassification probability:

[TABLE]

where $\hat{l}$ is the output of the maximum-likelihood classifier over the class-conditional densities described by $\mathbf{a}_{l}$. In this paper, we provide two asymptotic analyses of $P_{e}(\mathbf{a})$. First, we consider the diversity order, which characterizes the slope of $P_{e}(\mathbf{a})$ for a particular $\mathbf{a}$ as $\sigma^{2}\to 0$. Second, we consider the classification capacity, which characterizes the asymptotic error performance *averaged over * $\mathbf{a}$ as $n_{1},m_{1},n_{2},m_{2}$ go to infinity. For the latter case, we define a prior distribution over the matrix pairs $(\mathbf{A}_{l},\mathbf{B}_{l})$ in each class:

[TABLE]

where $a_{pq}$ is the $(p,q)$th element of matrix $\mathbf{A}$ and $b_{rs}$ is the $(r,s)$th element of matrix $\mathbf{B}$. Note that the column and row subspaces described by $\mathbf{A}$ and $\mathbf{B}$ are uniformly distributed over the Grassmann manifold because the matrix elements are i.i.d. Gaussian; however, the resulting K-S subspaces are not uniformly distributed.

II-B Diversity Order

For a fixed classification problem $\mathbf{a}$, the diversity order characterizes the decay of the misclassification probability as the noise power goes to zero. By analogy with the definition of the diversity order in wireless communications [35], we consider the asymptotic slope of $P_{e}(\mathbf{a})$ on a logarithmic scale as $\sigma^{2}\to 0$ that is the mismatch between data and model is vanishingly small. Formally, the diversity order is defined as

[TABLE]

In Section III, we characterize exactly the diversity order for almost every $\mathbf{a}$.

II-C Classification Capacity

The classification capacity characterizes the number of unique subspaces that can be discerned as $n_{1}$, $n_{2}$, $m_{1}$ and $m_{2}$ go to infinity. That is, we derive bounds on how fast the number of classes $L$ can grow as a function of signal dimension while ensuring the misclassification probability decays to zero almost surely. Here, we define a variable $m$111Note that $m$ is different from $M$, where $m$ is the variable we let to go to infinity and $M=m_{1}m_{2}$. and let it go to infinity. As $m$ grows to infinity we let the dimensions $m_{1}$, $m_{2}$, $n_{1}$ and $n_{2}$ scale linearly with $m$ as follows:

[TABLE]

for $\upsilon_{1},\upsilon_{2}\geq 1$ and $0\leq\kappa_{1},\kappa_{2}\leq 1$. We let the number of classes $L$ grow exponentially in $m$ as:

[TABLE]

for some $\rho\geq 0$, which we call the classification rate. We say that the classification rate $\rho$ is achievable if $\lim_{m\rightarrow\infty}E[P_{e}(\mathbf{a})]=0$. For fixed signal dimension ratios $\upsilon_{1},\upsilon_{2},\kappa_{1}$ and $\kappa_{2}$, we define $C(\upsilon_{1},\upsilon_{2},\kappa_{1},\kappa_{2})$ as the supremum over all achievable classification rates, and we call $C(\upsilon_{1},\upsilon_{2},\kappa_{1},\kappa_{2})$ (sometimes abbreviated by $C$) the classification capacity.

We can bound the classification capacity by the mutual information between the signal vector $\mathbf{y}$ and the matrix pair $(\mathbf{A}$, $\mathbf{B})$ that characterizes each Kronecker-structured class.

Lemma** 1****.**

The classification capacity satisfies:

[TABLE]

Where the mutual information is computed with respect to $p(\mathbf{a})$.

To prove lower bounds on the diversity order and classification capacity, we will need the following lemma, which gives the well-known Bhattacharyya bound on the probability of error of a maximum-likelihood classifier that chooses between two Gaussian hypotheses.

Lemma** 2**** ([36]).**

Consider a signal distributed according to $\mathcal{N}(\mu_{1},\Sigma_{1})$ or $\mathcal{N}(\mu_{2},\Sigma_{2})$ with equal priors. Then, define

[TABLE]

Supposing maximum likelihood classification, the misclassification probability is bounded by

[TABLE]

II-D Subspace Geometry

We characterize the subspace geometry in terms of principal angles. Principal angle defines as the canonical angles between elements of subspaces, and they induce a distance metric on the Grassmann manifold. If the principal angles between subspaces is large, this means that the subspaces are far apart and easily discernible.

Consider two linear subspaces $\mathcal{A}_{1}$ and $\mathcal{A}_{2}$ of $\mathbb{R}^{m}$ with same dimensions $n$ each. The principal angles between these two subspaces are defined recursively as follows:

[TABLE]

where $0\leq\theta_{1}\leq\theta_{2}\leq\cdots\leq\theta_{n_{1}}\leq\frac{\pi}{2}$ and the first principal angle $\theta_{1}$ is the smallest angle between all pairs of unit vectors in the first and the second subspaces [37].

The principal angles can be computed directly via computing the singular value decomposition (SVD) of $\mathbf{A}_{1}^{T}\mathbf{A}_{2}$, where $\mathbf{A}_{1}$ and $\mathbf{A}_{2}$ are orthonormal basis for the subspaces $\mathcal{A}_{1}$ and $\mathcal{A}_{2}$, respectively.

[TABLE]

where the cosine of principal angles, $\cos(\Theta_{A})=\text{diag}(\cos(\theta_{1}^{A}),\cos(\theta_{2}^{A}),\cdots,\cos(\theta_{n_{1}}^{A}))$, are the singular values of $\mathbf{A}_{1}^{T}\mathbf{A}_{2}$.

In this problem, suppose $\mathbf{A}_{1}$ and $\mathbf{A}_{2}$ are orthonormal basis for the subspaces $\mathcal{A}_{1}$ and $\mathcal{A}_{2}$ on which columns of signal approximately lies and $\mathbf{B}_{1}$ and $\mathbf{B}_{2}$ are orthonormal basis for the subspaces $\mathcal{B}_{1}$ and $\mathcal{B}_{2}$ on which rows of signal approximately lies. Then we define the orthonormal basis $\mathbf{D}_{1}=\mathbf{B}_{1}\otimes\mathbf{A}_{1}$ and $\mathbf{D}_{2}=\mathbf{B}_{2}\otimes\mathbf{A}_{2}$ for the Kronecker-structured subspaces $\mathcal{D}_{1}$ and $\mathcal{D}_{2}$, respectively. The cosine of principal angles between $\mathbf{D}_{1}$ and $\mathbf{D}_{2}$ are the singular values of $\mathbf{D}_{1}^{T}\mathbf{D}_{2}$ as follows:

[TABLE]

where the cosine of principal angle between two Kronecker subspaces is the Kronecker product of cosine of principal angles between two row subspaces and two column subspaces that is $\cos(\Theta)=\cos(\Theta_{A})\otimes\cos(\Theta_{B})$.

III Diversity Order

As mentioned in Section II, the diversity order measures how quickly misclassification probability decays with the noise power for a fixed number of discernible subspaces. By careful analysis using the Bhattacharrya bound, we derive an exact expression for the diversity order for almost every222With respect to the Lebesgue measure over $\mathcal{A}^{L}$. classification problem. First, we state an expression that holds in general.

Theorem 1**.**

For a classification problem described by the tuple $\mathbf{a}\in\mathcal{A}^{L}$ such that $r(\mathbf{A}_{l})=n_{1}$ and $r(\mathbf{B}_{l})=n_{2}$ for every $l$, the diversity order is $d(\mathbf{a})=r^{*}-n_{1}n_{2}$, where

[TABLE]

and where $r(\cdot)$ denotes the matrix rank.

Proof:

Applying the Bhattacharyya bound, the probability of a pairwise error between two Kronecker-structured classes $i$ and $j$ with covariances

[TABLE]

is bounded by

[TABLE]

where

[TABLE]

Using the well-known Kronecker product identities $(p\otimes q)\cdot(r\otimes s)=(pq\otimes rs)$ and $(p\otimes r)^{T}=(p^{T}\otimes r^{T})$ we can write the matrix $\mathbf{D}_{i}\mathbf{D}_{i}^{T}+\mathbf{D}_{j}\mathbf{D}_{j}^{T}$ as

[TABLE]

It is trivial that $r(A)=r(AA^{T})$, thus

[TABLE]

Let $\lambda_{i}$ and $\lambda_{j}$ denote the nonzero eigenvalues of $\mathbf{D}_{i}\mathbf{D}_{i}^{T}$ and $\mathbf{D}_{j}\mathbf{D}_{j}^{T}$ respectively, and let $\lambda_{ij}$ denote the nonzero eigenvalues of $\mathbf{D}_{i}\mathbf{D}_{i}^{T}+\mathbf{D}_{j}\mathbf{D}_{j}^{T}$ and $r^{*}_{ij}$ denote its rank. Then, we can write the pairwise bound in (17).

By construction,

[TABLE]

Using Weyl’s monotonicity theorem $2\lambda_{ijl}\geq\lambda_{il}$ and $2\lambda_{ijl}\geq\lambda_{jl}$ for every $1\leq l\leq n_{1}n_{2}$, Therefore,

[TABLE]

From this we can write

[TABLE]

Next, we bound $P_{e}(\mathbf{a})\leq\sum_{i\neq j}P_{e}(\mathbf{D}_{i},\mathbf{D}_{j})$ via the union bound. For all the $L$ subspaces, we obtain the pairwise error probability and by invoking the union bound over all the subspaces we obtain:

[TABLE]

Taking logarithm on both sides we obtain:

[TABLE]

Putting this and (21) into the definition of the diversity order from (8), we obtain

[TABLE]

Finally, [36] shows that the Bhattacharyya bound is exponentially tight as the pairwise error decays to zero. Furthermore, the union bound is exponentially tight. Therefore, the above inequality holds with equality, and $d(\mathbf{a})~{}=~{}r^{*}~{}-~{}n_{1}n_{2}.$ ∎

For almost every classification problem, the rank $r^{*}$ has the same value, as we show in the next lemma.

Lemma** 3****.**

For almost every classification problem $\mathbf{a}$, the matrices $\begin{bmatrix}\mathbf{B}_{i}\otimes\mathbf{A}_{i}&\mathbf{B}_{j}\otimes\mathbf{A}_{j}\end{bmatrix}$ have rank

[TABLE]

where $[\cdot]^{+}$ denotes the positive part of a number.

Proof:

Using standard matrix properties (e.g., [38]), we can write

[TABLE]

Applying Lemma 4 from Appendix B, we obtain

[TABLE]

Almost every matrix has full rank, so $r(\mathbf{B}_{i}\otimes\mathbf{A}_{i})=r(\mathbf{B}_{j}\otimes\mathbf{A}_{j})=n_{1}n_{2}$ almost everywhere, so we can rewrite (28) as

[TABLE]

Next, we study the three possible cases for (29).

Case 1: $n_{2}<m_{2}<2n_{2}$ and $n_{1}\leq\frac{m_{1}}{2}$. Here,

[TABLE]

Case 2: $n_{2}\leq\frac{m_{2}}{2}$ and $n_{1}<m_{1}<2n_{1}$. Here,

[TABLE]

Case 3: $n_{2}<m_{2}<2n_{2}$ and $n_{1}<m_{1}<2n_{1}$. Here,

[TABLE]

where the first and second equalities for each case hold almost everywhere, and the third equality for each case follows from Lemma 4. Combining the three cases yields the claim. ∎

Applying Lemma 3 to Theorem 1, an exact expression for the diversity order follows immediately.

Corollary 1**.**

For almost every classification problem $\mathbf{a}$, the diversity order is

[TABLE]

III-A Diversity Order Gap

Diversity order characterize the slope of error probability, higher the diversity order faster the decay of misclassification probability. Since the Kronecker-structured subspaces comes from a restricted set of subsapces, the error performance of these subspaces can be worse. Therefore, to verify the efficiency of Kronecker subspaces, we characterizes the diversity order gap as the difference between the slope of misclassification probability of K-S subspaces and the standard subspaces. This diversity order gap is a function of signal dimensions, that is, $n_{1},n_{2},m_{1}$ and $m_{2}$. We derive the signal dimension regimes where the diversity order gap is significant or/and zero.

Diversity order for K-S subspaces:

[TABLE]

For the standard subspaces model in (3), the signal of interest $\mathbf{Y}\in\mathbb{R}^{M}$ and coefficient vector $\mathbf{X}\in\mathbb{R}^{N}$ where, $M=m_{1}m_{2}$ and $N=n_{1}n_{2}$. From [32], for the standard subspaces of same dimensions the diversity order would look like $N-[2N-M]^{+}$. This can be written in terms of Kronecker signal dimensions.

Diversity order for standard subspace:

[TABLE]

We observe that the diversity order for K-S models is never greater to the diversity order of standard subspace, for any value of $n_{1},n_{2},m_{1},m_{2}$. However, for some regimes the diversity order of K-S model is smaller or equal to standard subspaces.

When $n_{1}<m_{1}<2n_{1}$ and $n_{2}<m_{2}<2n_{2}$

if $m_{1}m_{2}>2n_{1}n_{2}$ then $d_{\text{K-S}}<d_{\text{STD}}$:

[TABLE] 2. 2.

if $m_{1}m_{2}<2n_{1}n_{2}$ then $d_{\text{K-S}}<d_{\text{STD}}$:

[TABLE]

where $\gamma=d_{\text{STD}}-d_{\text{K-S}}$ is the diversity order gap. For any other region no diversity order gap exists, that is, $d_{\text{K-S}}=d_{\text{STD}}$. The details are provided in Appendix A.

The high-SNR classification performance of K-S subspaces is the same as general subspaces when the subspace dimensions are small, even though K-S subspaces are structured, involve fewer parameters, and are easier to train.

III-B Misclassification Probability in terms of Row and Column Subspaces Geometry

We derive a more accurate and tight high-SNR approximation of the probability of error in terms of principal angles between the K-S subspaces and also in terms of principal angle between the individual rows and columns subspaces. Using the eigenvalue decomposition of covariance of row subspace $\mathbf{A}_{i}\mathbf{A}_{i}^{T}=\mathbf{U}_{i}^{A}\lambda_{i}^{A}(\mathbf{U}_{i}^{A})^{T}$ and the column subspace $\mathbf{B}_{i}\mathbf{B}_{i}^{T}=\mathbf{U}_{i}^{B}\lambda_{i}^{B}(\mathbf{U}_{i}^{B})^{T}$, where $\mathbf{U}_{i}^{A}\in\mathbb{R}^{m_{1}\times n_{1}},\mathbf{U}_{i}^{B}\in\mathbb{R}^{m_{2}\times n_{2}}$ are the orthonormal basis of row and column subspace respectively and the $\mathrm{diag}(\lambda_{i}^{A})\in\mathbb{R}^{n_{1}},\mathrm{diag}(\lambda_{i}^{B})\in\mathbb{R}^{n_{2}}$ are the eigenvalues of row and column subspaces, we can write the signal covariance as:

[TABLE]

Similarly, $\mathbf{D}_{j}\mathbf{D}_{j}^{T}=\mathbf{U}_{j}\lambda_{j}\mathbf{U}_{j}^{T}$. From [39], the Kronecker product of two orthonormal matrix is a orthonormal matrix, thus $\mathbf{U}_{i},\mathbf{U}_{j}\in\mathbb{R}^{m_{1}m_{2}\times n_{1}n_{2}}$ are the orthonormal bases and the diagonal elements of $\lambda_{i},\lambda_{j}\in\mathbb{R}^{n_{1}n_{2}\times n_{1}n_{2}}$ are the eigenvalues. From equation (27), the rank of sum of two Kronecker products is written as:

[TABLE]

Since the intersection of two K-S subspaces define this rank and hence plays an important role in bounding the misclassification probability from above. According to [33], one can write the covariances of K-S subspaces in terms of subspaces intersections as follows:

[TABLE]

Here $\mathbf{U}_{i,\cap},\,\,\mathbf{U}_{j,\cap}\in\mathbf{R}^{m_{1}m_{2},r_{\cap}}$ corresponds to the K-S subspace intersection and $\mathbf{U}_{i,\setminus},\,\,\mathbf{U}_{j,\setminus}\in\mathbf{R}^{m_{1}m_{2},n_{1}n_{2}-r_{\cap}}$ corresponds to the set minus $\mathcal{D}_{i}\setminus\mathcal{D}_{j}$ and $\mathcal{D}_{j}\setminus\mathcal{D}_{i}$ respectively. Here $r_{\cap}$ accounts for the overlap between the subspaces, smaller the overlap between subspaces easier it to discern the classes. While on the other hand, $r_{\cap}=n_{1}n_{2}$ means the complete overlap between subspaces and it becomes hard to discriminate between classes.

Theorem 2**.**

As $\sigma^{2}\rightarrow 0$, the misclassification probability in terms of principal angle between individual row and column subspaces is upper bounded as

[TABLE]

where

[TABLE]

$t_{1}=\lfloor\frac{(n_{2}-n_{1})-\sqrt{(n_{2}-n_{1})^{2}+4r_{\cap}}}{2}\rfloor$*,

$t_{2}=\lfloor\frac{(n_{1}-n_{2})-\sqrt{(n_{1}-n_{2})^{2}+4r_{\cap}}}{2}\rfloor$ and $\mathrm{pdet}$ denotes the pseudo-determinant.*

Proof:

Appendix C. ∎

In case of no overlap between subspaces, that is, $r_{\cap}=0$, both $t_{1}=t_{2}=0$ and as the misclassification probability is inversely related to the product of all $n_{1}n_{2}$ principal angles, this makes the misclassification error negligibly small. On the other side, with subspace overlap $r_{\cap}\neq 0$, $t_{1}$ and $t_{2}$ has some positive value, there exists some non-trivial principal angles which effect the classification performance and it becomes very hard to distinguish between the subspaces.

IV Classification capacity

In this section, we derive upper and lower bounds on the classification capacity that hold approximately for large $\sigma^{2}$. Detailed analysis can be found in the long version of the paper.

Theorem 3**.**

The classification capacity is upper bounded by

[TABLE]

and

[TABLE]

Proof:

The upper bound follows from an upper bound on the mutual information $I(\mathbf{y};\mathbf{A},\mathbf{B})=h(\mathbf{y})-h(\mathbf{y}|\mathbf{A},\mathbf{B})$ between the dictionary pairs $(\mathbf{A},\mathbf{B})$ and the signal $\mathbf{y}$ and invoke Lemma 1. In particular,

[TABLE]

Given the conditional distribution $p(\mathbf{y}|(\mathbf{B}\otimes\mathbf{A}))=\mathcal{N}(0,(\mathbf{B}\otimes\mathbf{A})(\mathbf{B}\otimes\mathbf{A})^{T}+\sigma^{2}\cdot\mathbf{I})$ and following the analysis similar to [32] we bound the conditional entropy as:

[TABLE]

From the i.i.d. Gaussian outer bound on entropy, we can derive a naive bound on the marginal entropy:

[TABLE]

Now consider the case when both $\mathbf{A}$ and $\mathbf{B}$ are tall i.e. $m_{1}>n_{1}$ and $m_{2}>n_{2}$. Further suppose that $n_{1}<m_{2}$. Then, we can derive a tighter outer bound on $h(\mathbf{y})$. Let $\mathbf{y}_{p}$ be the first $n_{1}$ columns of $\mathbf{y}$ and let $\mathbf{y}^{\prime}_{p}$ be the rest $m_{2}-n_{1}$ columns of $\mathbf{y}$. Then, $\mathbf{y}^{\prime}_{p}\in\mathbb{R}^{m_{1}\times(m_{2}-n_{1})}$, and we can derive the following high-SNR approximation on $h(\mathbf{y})$:

[TABLE]

[TABLE]

Now, let $\mathbf{y}_{q}$ be the first $n_{2}$ columns of $\mathbf{y}$ and let $\mathbf{y}^{\prime}_{q}$ denotes the rest $m_{1}-n_{2}$ columns of $\mathbf{y}$. Then, $\mathbf{y}^{\prime}_{p}\in\mathbb{R}^{(m_{2}-n_{2})\times m_{2}}$, and we derive the following high-SNR approximation on $h(\mathbf{y})$:

[TABLE]

[TABLE]

Combining (43) and (45), we obtain the differential entropy:

[TABLE]

From (39) and (47), as $m\rightarrow\infty$ we can find the bound.

∎

Proof:

In order to obtain the lower bound on classification capacity we apply the Bhatacharyya bound on probability of pairwise error between two Kronecker-subspaces $i$ and $j$. By expanding $r^{*}_{ij}$ in (22) and bounding the value of $\lambda_{ij\{2n_{1}n_{2}-[2n_{1}-m_{1}]^{+}[2n_{2}-m_{2}]^{+}\}}$ away from zero as $m\rightarrow\infty$. If

[TABLE]

then surely $P_{e}(\mathbf{a})$ goes to zero as $m\rightarrow\infty$. ∎

To compare the upper and lower bounds, consider the symmetric case, i.e. $m_{1}=m_{2}=m$ and $n_{1}=n_{2}=n$ and $m>n$. The gap between the prelog factor of the upper and lower bounds is $(m-n)^{2}$ and we leave tightening these bounds as future work.

V Kronecker-Structured Learning of Discriminative Dictionaries (K-SLD2)

Here we introduce K-SLD2, an efficient and effective method for learning discriminative dictionary pairs for classifying two-dimensional signals $~{}{\mathbf{Y}\in\mathbf{R}^{m_{1}\times m_{2}}}$ in (2). For $L$ number of classes let $K$ is the number of training samples per class. We define $Y_{i}$ as a collection of $K$ 2-D signals corresponding to class $i$. That is,

[TABLE]

for $i=1,\cdots,L$ and $\mathbf{Y}_{ji}\in\mathbf{R}^{m_{1}\times m_{2}}$ is the $j$th signal belonging to class $i$.

We suppose that each class corresponds to a different subspace. Thus, our objective is to learn the structured dictionary pairs $\mathbf{\mathbb{A}}=\{\mathbf{A}_{1},\mathbf{A}_{2},\cdots,\mathbf{A}_{L}\}$ and $\mathbf{\mathbb{B}}=\{\mathbf{B}_{1},\mathbf{B}_{2},\cdots,\mathbf{B}_{L}\}$ that describe the training data. We define the set of structured dictionary pairs as $(\mathbf{\mathbb{A}},\mathbf{\mathbb{B}})=\{(\mathbf{A}_{1},\mathbf{B}_{1}),(\mathbf{A}_{2},\mathbf{B}_{2}),\cdots,(\mathbf{A}_{L},\mathbf{B}_{L})\}$, where $(\mathbf{A}_{i},\mathbf{B}_{i})$ is the class-specific sub-dictionary pair associated with class $i$.

Let $\mathbf{\mathbb{X}}=\{S_{1},S_{2},\cdots,S_{L}\}$ be a set of coefficient matrices for each signal, where $S_{i}=\{X_{1i},X_{2i},\dots,X_{Ki}\}$ is the sub-matrix containing the coefficients of all the training samples $Y_{i}$ belongs to a class $i$ over the dictionary pair $(\mathbf{\mathbb{A}},\mathbf{\mathbb{B}})$. We write, $X_{ji}=\{\mathbf{X}_{ji}^{1},\mathbf{X}_{ji}^{2},\cdots,\mathbf{X}_{ji}^{L}\}$ a representation of signal $j$ of class $i$ over the dictionary pair $(\mathbf{\mathbb{A}},\mathbf{\mathbb{B}})$, where $\mathbf{X}_{ji}^{l}\in\mathbb{R}^{n_{1}\times n_{2}}$ is the coefficient of a training sample $\mathbf{Y}_{ji}$ over the dictionary pair $(\mathbf{A}_{k},\mathbf{B}_{k})$. That is, $(\mathbf{\mathbb{A}},\mathbf{\mathbb{B}})$ represent an overcomplete dictionary, and we learn coefficients such that

[TABLE]

Algorithm Description: We want the dictionaries to have both high reconstruction power and high discriminative power. To encourage discriminability, we want a signal $\mathbf{Y}_{i}$ to be well represented by the class-specific dictionary $(\mathbf{A}_{i},\mathbf{B}_{i})$, and (comparatively) poorly represented by the other dictionaries $(\mathbf{A}_{l},\mathbf{B}_{l}),l\neq i$. Here, $\mathbf{A}_{l}\mathbf{X}_{ji}^{l}\mathbf{B}_{l}^{T}$ denotes the representation of the training sample $\mathbf{Y}_{ji}$ over the $l$th dictionary pair. Then, the dictionaries discriminate well if $||\mathbf{Y}_{ji}-\mathbf{A}_{l}\mathbf{X}_{ji}^{l}\mathbf{B}_{l}^{T}||_{2}$ is small for $i=l$ and large for $i\neq l$. This leads to a optimization problem:

[TABLE]

The first term in (50) encourages the representation power of the joint, overcomplete dictionary, whereas the second and third terms encourage the discrimination power of the class-specific dictionaries. This problem is jointly nonconvex, but it is convex in the individual variables $\mathbb{A},\mathbb{B},\mathbb{X}$ when the other are fixed. We solve (50) by alternating between the variables, solving the individual convex problem, and iterating until convergence. Thus, we divide (50) into three subproblems: updating $\mathbb{X}$ while fixing $\mathbb{A}$ and $\mathbb{B}$; updating $\mathbb{A}$ while fixing $\mathbb{X}$ and $\mathbb{B}$; and updating $\mathbb{B}$ while fixing $\mathbb{X}$ and $\mathbb{A}$. Each subproblem further has a closed-form solution. The solution to the first subproblem is

[TABLE]

Then, the solution to the second subproblem is

[TABLE]

Finally, the solution to the third subproblem is, for $i=l$

[TABLE]

and for $i\neq l$:

[TABLE]

These iterations continue until changes in the objective function are sufficiently small.

Convergence: This procedure is guaranteed to converge in terms of the objective function value via the following argument. Because each subproblem is convex, the value of the objective function is nondecreasing as iterations proceed. Furthermore, because the objective function is bounded below, the nondecreasing sequence of function values must converge. A sample trajectory is shown in Fig. 1(a). Here, K-SLD2 is trained on the extended YaleB dataset. The overall reconstruction error is shown in (50), whereas Fig. 1(b), shows both that the signal $Y_{i}$ is well represented by the dictionary pair $(\mathbf{A}_{i},\mathbf{B}_{i})$ and as the number of iterations increases the other dictionary pairs $(\mathbf{A}_{l},\mathbf{B}_{l}),l\neq i$ start loosing their ability to represent $Y_{i}$.

Classification Procedure: Given a test signal $\mathbf{Y}$ to classify, we first find the coefficient matrices for each class using

[TABLE]

This problem is convex and has a closed-form solution. Then, we compute the reconstruction error for each class-specific dictionary:

[TABLE]

Finally, we make the prediction $\hat{k}=\arg\min_{i=1,\cdots,L}(e_{i})$; i.e., the class with the smallest reconstruction error.

Computational Complexity: In this analysis we use the fact that: 1) if $\mathbf{A}\in\mathbb{R}^{m_{1}\times n_{1}}$ and $\mathbf{X}\in\mathbf{R}^{n_{1}\times n_{2}}$ then the matrix multiplication $\mathbf{A}\mathbf{X}$ has complexity $m_{1}n_{1}n_{2}$. 2) if a non singular matrix $\mathbf{A}\in\mathbb{R}^{n_{1}\times n_{1}}$, then $\mathbf{A}^{-1}$ has complexity $n_{1}^{3}$. We obtain a complexity (in terms of matrix multiplications and additions) of

[TABLE]

If we assume $m_{1}=m_{2}=\sqrt{m}$ and $n_{1}=n_{2}=\sqrt{n}$ then the complexity becomes

[TABLE]

Which is a reduction when compared to standard subspace learning with computational complexity of $\mathcal{O}(KLnm).$

VI Numerical Results

In this section, we evaluate first demonstrate that the empirical classification performance, when the classes are perfectly known, agrees with the diversity order and bounds derived above. Then, we demonstrate the learning and classification performance of K-SLD2 on both synthetic and real-world data.

VI-A Diversity Order

VI-A1 Synthetic Data

We randomly choose two classes by drawing matrix pairs $\mathbf{A}_{i}$ and $\mathbf{B}_{i}$ independently from the distribution in (7). Then, we draw data samples i.i.d. from the class-conditional densities in (3). We classify each data sample by minimizing the Mahalanobis distance associated with the covariance of each class-conditional density. We consider five cases, in which we fix $m_{1}=m_{2}=m$ and vary $n_{1}$ and $n_{2}$. In Fig.2 we plot the misclassification probability $P_{e}$ against the SNR in dB, averaged over $10^{5}$ random draws from each class. We also plot the upper bound on misclassification probability in terms of principal angles for each case described in (LABEL:Eq:principal). Where dotted colored line shows the misclassification probability associated with the corresponding solid line for each case. In each case, the empirical performance agrees with the diversity predictions with an offset. This offset is large when the ambient signal dimension is small and with large dimensions this offset approaches to zero.

VI-B Dictionary Learning Algorithm

In this section we evaluate the performance of K-SLD2 algorithm on synthetic data and two real world datasets: extended YaleB face dataset [40] and the UCI EEG dataset [41], which differentiates the EEG signals of control patients and those who suffer from alcoholism. We compare the performance to state-of-the-art dictionary learning methods such as FDDL [26], DLSI [13], LRSDL [27], standard subspace learning (SSL) using (1) as a baseline method, and the standard kernel support vector machine (SVM). We perform learning and classification on unprocessed signals. When appropriate, we choose model hyper-parameters via cross-validation.

VI-B1 Synthetic data

We consider two class classification problem where we draw two matrix pairs $\mathbf{A}_{i}$ and $\mathbf{B}_{i}$ independently from (7) and draw data samples i.i.d from the class-conditional densities in (3). For this experiment we choose the dimensions of the signal to be $32\times 32$ which lies on the row and column subspaces of dimension $13$ and $17$, respectively. For each class we draw 10 samples for training/dictionary learning and 50 samples for testing. In total we have 60 samples per class. For learning K-S dictionaries using K-SLD2 we use the 2-D signal as it is while for the other learning algorithms we first vectorize the signal (dimension $1024\times 1$). Fig. 3 compares the performance of learned dictionaries using different methods as the SNR decreases. When the noise power is low, that is, $\leq 10^{1}$, standard subspace learning and K-SLD2 performs equally well, but as the noise power increases a significant gain in performance is observed as evident in Fig. 3. We find best classification performance for SVM with polynomial kernel of degree 3.

VI-B2 Face Recognition

The extended YaleB dataset consists of 2,414 frontal face images from 38 individuals captured under varying lighting conditions. For each class, we use 10 images for training/dictionary learning and the remaining 54 images for testing. In Figure 4 we show the dictionaries learned by K-SLD2 vs. a standard subspace learning model, and we observe that the standard model learns dictionary atoms that look similar to a few reference faces for each class, whereas the K-SLD2 learns more abstract dictionary atoms. This is in part due to imposition of the Kronecker structure on the dictionary atoms, as well as the larger number of atoms possible in a K-S dictionary.

The best hyper-parameters for K-SLD2 turn out to be $n_{1}=13$, $n_{2}=17$, and $\mu=0.9$. For standard subspace model, we obtain the best classification accuracy for $10$ dictionary atoms. The K-SLD2 uses more atoms overall, but each atom is described by fewer parameters. In Table I, we compare the classification accuracy of K-SLD2 with the other dictionary learning methods. K-SLD2 offers better performance in this case, rather close to FDDL and correctly classify 11.16% of the images than the baseline method. Furthermore, K-SLD2 learns a much more compact model, needing on the order of $1/10$th of the parameters of any other method.

We also calculate the normalized reconstruction error (NRE) for all the learning algorithms as follows:

[TABLE]

where $\mathbf{Y}$ is the signal of interest and $\mathbf{\hat{Y}}$ is the reconstructed signal. Table I shows that K-SLD2 provides the smallest NRE, reducing the error by 38.19% over the baseline. Finally, we observe that the computational complexity, measured in training runtime on a standard desktop computer, is small. LRSDL method requires 50.51 seconds for training while K-SLD2 model requires only 0.11 seconds.

We show the classification and representation performance as a function of the size of the training set in Figs. 5(a) and 5(b), respectively. When the number of samples for training is very small, say 10 samples per class, K-SLD2 model performance is superior, owing in part to the compact model. However, other methods outperform K-SLD2 as the number of samples increases. On the other hand, the reconstruction error of K-SLD2 model is always smaller than other methods for any number of training samples as evident in Fig. 5(b). In Fig. 7, we show a subset of raw YaleB face images used for the reconstruction and classification and compare the performance of K-SLD2 with SSL, where face in white box are the ones with the wrong label prediction.

VI-B3 EEG Dataset

We evaluate the performance of K-SLD2 on the UCI EEG dataset [41], where EEG from the brain were recorded by placing the 64 electrodes on the scalp sampled at 256 Hz for 1 second to examine the correlation of EEG signal to an individual’s alcoholism. Here, we obtain a 2-D signal with electrodes on one axis and the corresponding electrical signal time series on the other. This classification problem is analogous to binary classification having two categories of individuals either belongs to alcoholism or controlled group. The full datasets contains 120 trials for 122 subjects. Similar to YaleB face recognition dataset, we use 10 signals per class for training/dictionary learning and the remaining images for testing and find the value of $n_{1}=10$ and $n_{2}=6$ using cross-validation.

We compare the performance of K-SLD2 in Table II with other dictionary learning methods. Again, K-SLD2 gives better classification performance and requires very few model parameters. In terms of NRE, K-SLD2 reconstruction error is less than 41% of the best among the other methods. We obtain this performance gain for K-SLD2 because the dictionaries with separable structure are very good at signal representation [30]. Similarly, we plot the classification and reconstruction accuracy in Figs. 6(a) and 6(b), respectively. Here, by contrast to YaleB, we observe competitive performance for a larger number of training samples, due perhaps to the explicitly multidimensional nature of EEG signals. Reconstruction performance, measured in NRE, remains superior to other methods.

VII Conclusion

We derive the performance limits on the classification performance of Kronecker-structured models. We derive an exact expression for the slope of misclassification probability as the noise power goes to zero. In high SNR regime, we derive a more accurate and tighter bound on misclassification probability which is determined by the product of principal angles between Kronecker subspaces. We determine the upper and lower bounds on the rate at which the number of classes can grow as the signal dimension goes to infinity. We have also proposed a dictionary learning algorithm K-SLD2, for fast classification and compact representation of multidimensional signals. This algorithm balances the learning of class-specific, Kronecker-structured subspaces against the learning of an general overcomplete dictionary that allows for the representation of general signals. Finally we show that K-SLD2 has improved classification performance over state-of-the-art dictionary learning methods, especially when the size of the training set is small, and competitive reconstruction performance in general.

Appendix A Diversity order gap

Given the K-S diversity order $d_{\text{K-S}}$ and the standard subspace diversity order $d_{\text{STD}}$. We derive the diversity gap

[TABLE]

in terms of signal dimensions for different regions:

Region 1: $m_{1}>2n_{1}$ and $m_{2}>2n_{2}$

Since $m_{1}>2n_{1}$ and $m_{2}>2n_{2}$ therefore, ${[2n_{1}-m_{1}]^{+}=}~{}[2n_{2}-m_{2}]^{+}=0$ and $m_{1}m_{2}>4n_{1}n_{2}$ makes $[2n_{1}n_{2}-m_{1}m_{2}]^{+}=0$.

Region 2: $n_{1}<m_{1}<2n_{1}$ and $m_{2}>2n_{2}$

Since $m_{2}>2n_{2}$ therefore, $[2n_{2}-m_{2}]^{+}=0$ also $m_{1}>n_{1}$ and $m_{2}>2n_{2}$ implies $m_{1}m_{2}>2n_{1}n_{2}$ therefore,  $[2n_{1}n_{2}-~{}m_{1}m_{2}]^{+}=~{}0$.

Region 3: $n_{2}<m_{2}<2n_{2}$ and $m_{1}>2n_{1}$

Using the similar argument the diversity gap is 0.

Region 4: $n_{1}<m_{1}<2n_{1}$ and $n_{2}<m_{2}<2n_{2}$

Since $n_{1}<m_{1}<2n_{1}$ and $n_{2}<m_{2}<2n_{2}$ implies that $n_{1}n_{2}<m_{1}m_{2}<4n_{1}n_{2}$, this gives rise to two different subregions which are $m_{1}m_{2}>2n_{1}n_{2}$ and $m_{1}m_{2}<2n_{1}n_{2}$.

For $m_{1}m_{2}>2n_{1}n_{2}$, $n_{1}<m_{1}<2n_{1}$ and $n_{2}<m_{2}<2n_{2}$ which implies $[2n_{1}n_{2}-m_{1}m_{2}]^{+}=0$ we derive the diversity order gap as:

[TABLE]

On the other hand if $m_{1}m_{2}<2n_{1}n_{2}$, $n_{1}<m_{1}<2n_{1}$ and $n_{2}<m_{2}<2n_{2}$,

[TABLE]

Appendix B Intersection of Kronecker Subspaces

Here we characterize the dimension of intersections of subspaces spanned by Kronecker products of matrices. To the best of our knowledge this result is not in the literature, although its statement is intuitive.

Lemma** 4****.**

Suppose $\dim[\mathcal{R}(A_{i})\bigcap\mathcal{R}(A_{j})]~{}=~{}x$ and $\dim[\mathcal{R}(B_{i})\bigcap\mathcal{R}(B_{j})]~{}=~{}y$, where $\mathcal{R}(\cdot)$ denotes the range space of a matrix. Then,

[TABLE]

Proof:

From [39, p. 447] for $p\in\mathbb{R}^{m_{1}\times n_{1}}$ and $q\in\mathbb{R}^{m_{2}\times n_{2}}$, we have

[TABLE]

Therefore, we can write the dimension as

[TABLE]

Rearranging terms, we obtain

[TABLE]

Next, let $\mathbf{A}_{ij}$ and $\mathbf{B}_{ij}$ be matrices whose column spans are $\mathcal{R}(\mathbf{A}_{i})\bigcap\mathcal{R}(\mathbf{A}_{j})$ and $\mathcal{R}(\mathbf{B}_{i})\bigcap\mathcal{R}(\mathbf{B}_{j})$, respectively. It is straightforward to verify that

[TABLE]

and

[TABLE]

Therefore, we can rewrite the subspace dimension as

[TABLE]

Next, we can apply the lemma of [39, p. 447] in reverse, yielding

[TABLE]

∎

Appendix C Proof of Theorem 2

Expanding the Bhattacharyya bound from (15) we obtain the misclassification probability bound in terms of $\lambda_{ij}$ the nonzero eigenvalues of $\mathbf{D}_{i}\mathbf{D}_{i}^{T}+\mathbf{D}_{j}\mathbf{D}_{j}^{T}$ in (18) as:

[TABLE]

Now, our aim is to expand $\prod_{l=1}^{r^{*}_{ij}}\lambda_{ijl}$ in terms of principal angles.

[TABLE]

As the image of $U_{i,\cap}$ is orthogonal to $U_{i,\setminus}$ we can write:

[TABLE]

Following few simple mathematical steps as described in [33] we obtain:

[TABLE]

By expanding $U_{i,\cap},U_{j,\cap},U_{i,\setminus},U_{j,\setminus}$ in terms of their row and columns subspace Kronecker products and then following some simple Kronecker product properties we obtain:

[TABLE]

By careful inspection of $\mathrm{det}(U_{j,\setminus}^{AT}U_{i,\setminus}^{A}U_{i,\setminus}^{AT}U_{j,\setminus}^{A})$ we find that product of eigenvalues of $(U_{j,\setminus}^{AT}U_{i,\setminus}^{A})(U_{j,\setminus}^{AT}U_{i,\setminus}^{A})^{T}$ is the square of the singular values of $(U_{j,\setminus}^{AT}U_{i,\setminus}^{A})$ and are the cosines square of the principal angles between the subspaces. Therefore we obtain:

[TABLE]

In terms of principal angles of individual row and column subspaces we obtain:

[TABLE]

Substituting this in (64), we obtain the desired results as stated in Theorem 2.