TL;DR
This paper introduces a novel approach for learning sparse tensor representations through mixtures of separable dictionaries, providing theoretical guarantees and efficient algorithms for both batch and online learning.
Contribution
It generalizes separable dictionary learning by proposing mixture models, deriving identifiability conditions, and developing algorithms for practical tensor data analysis.
Findings
Proposed mixture dictionary model improves tensor data representation.
Derived conditions ensure local identifiability of the learned dictionaries.
Algorithms demonstrate effectiveness in numerical experiments.
Abstract
This work addresses the problem of learning sparse representations of tensor data using structured dictionary learning. It proposes learning a mixture of separable dictionaries to better capture the structure of tensor data by generalizing the separable dictionary learning model. Two different approaches for learning mixture of separable dictionaries are explored and sufficient conditions for local identifiability of the underlying dictionary are derived in each case. Moreover, computational algorithms are developed to solve the problem of learning mixture of separable dictionaries in both batch and online settings. Numerical experiments are used to show the usefulness of the proposed model and the efficacy of the developed algorithms.
| Notation | Definition | Notation | Definition |
|---|---|---|---|
| , | , | ||
| Covering number of set w.r.t. norm | Separation rank of matrix | ||
| Oblique manifold in | Euclidean unit sphere in | ||
| Set of LSR matrices: | |||
| with : Set of KS matrices on | with | ||
| Closure of | |||
| Compact constraint set in LSR-DL problem: | |||
| one of , , , or | |||
| Unstructured | KS-DL () | LSR-DL () | ||||||
| Image | Noise | -SVD [5] | SeDiL [11] | BCD [29] | TeFDiL | BCD | STARK | TeFDiL |
| House | 35.6697 | 23.1895 | 31.6089 | 36.2955 | 32.2952 | 33.4002 | 37.1275 | |
| 25.4684 | 23.6916 | 24.8303 | 27.5412 | 21.6128 | 27.3945 | 26.5905 | ||
| Castle | 33.0910 | 23.6955 | 32.7592 | 34.5031 | 30.3561 | 37.0428 | 35.1000 | |
| 22.4184 | 23.2658 | 22.3065 | 24.6670 | 20.4414 | 24.4965 | 23.3372 | ||
| Mushroom | 34.4957 | 25.8137 | 33.2797 | 36.5382 | 32.2098 | 36.9443 | 37.7016 | |
| 22.5495 | 22.9464 | 22.8554 | 22.9284 | 21.7792 | 25.1081 | 22.8374 | ||
| Lena | 33.2690 | 23.6605 | 30.9575 | 34.8854 | 31.1309 | 33.8813 | 35.3009 | |
| 22.5070 | 23.4207 | 21.6985 | 23.4988 | 19.5989 | 24.8211 | 23.1658 | ||
| Image | Noise | -SVD | |||||
| Mushroom | 36.5382 | 36.7538 | 37.4173 | 37.4906 | 37.7016 | 34.4957 | |
| 22.9284 | 22.8352 | 22.8384 | 22.8419 | 22.8374 | 22.5495 | ||
| Number of parameters | 265 | 1060 | 2120 | 4240 | 8480 | 147456 | |
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Learning Mixtures of Separable Dictionaries for Tensor Data: Analysis and Algorithms
Mohsen Ghassemi, Zahra Shakeri,
Anand D. Sarwate, and Waheed U. Bajwa M. Ghassemi, A.D. Sarwate, and W.U. Bajwa are with the Department of Electrical and Computer Engineering, Rutgers, The State University of New Jersey, Piscataway, NJ, 08854 USA (Emails: {m.ghassemi, anand.sarwate, waheed.bajwa}@rutgers.edu). Z. Shakeri was with the Department of Electrical and Computer Engineering, Rutgers, The State University of New Jersey, Piscataway, NJ, 08854 USA and is now with Electronic Arts (Email: [email protected])Some of the results reported here were presented at CAMSAP 2017 [1] and ISIT 2019 [2]. This work is supported in part by the U.S. National Science Foundation under awards CCF-1453073 and CCF-1910110 and by the U.S. Army Research Office under award W911NF-17-1-0546.
Abstract
This work addresses the problem of learning sparse representations of tensor data using structured dictionary learning. It proposes learning a mixture of separable dictionaries to better capture the structure of tensor data by generalizing the separable dictionary learning model. Two different approaches for learning mixture of separable dictionaries are explored and sufficient conditions for local identifiability of the underlying dictionary are derived in each case. Moreover, computational algorithms are developed to solve the problem of learning mixture of separable dictionaries in both batch and online settings. Numerical experiments are used to show the usefulness of the proposed model and the efficacy of the developed algorithms.
Index Terms:
Dictionary learning, Kronecker structure, sample complexity, separation rank, tensor rearrangement.
I Introduction
Many data processing tasks such as feature extraction, data compression, classification, signal denoising, image inpainting, and audio source separation use sparse representations learned from data [3, 4, 5]. In many cases, these applications also involve data samples that are naturally structured as multiway arrays, also known as multidimensional arrays or tensors. Instances of multidimensional or tensor data include videos, hyperspectral images, tomographic images, and multiple-antenna wireless channels. Despite the ubiquity of tensor data in many applications, traditional data-driven sparse representation approaches disregard their multidimensional structure. This can result in sparsifying models with a large number of parameters. With the increasing availability of large data sets, it is crucial to keep sparsifying models reasonably small to ensure their scalable learning and efficient storage within devices such as smartphones and drones.
Our focus in this paper is on learning of “compact” models that yield sparse representations of tensor data. To this end, we study dictionary learning (DL) for tensor data. The goal in DL, which is an effective and popular data-driven technique for obtaining sparse representations of data [3, 5, 4], is to learn a dictionary such that every data sample can be approximated by a linear combination of a few atoms (columns) of . While DL has been widely studied, traditional DL approaches flatten tensor data and then employ methods designed for vector data [5, 6]. This ignores the multidimensional structure in tensor data, resulting in dictionaries with a large number of parameters. One intuitively expects that dictionaries which exploit the correlation and structure across tensor modes will have fewer parameters, leading to improvements in storage requirements, computational complexity, and generalization performance, especially when training data are noisy or scarce.
To reduce the number of parameters in dictionaries for tensor data, and to better exploit the correlation among different tensor modes, some recent DL works use tensor decompositions such as the Tucker decomposition [7] and CANDECOMP/PARAFAC decomposition (CPD) [8] for learning of “structured” dictionaries. The idea in structured DL for tensor data is to restrict the class of dictionaries during training to the one imposed by the tensor decomposition under consideration [9]. For example, structured DL based on the Tucker decomposition of -way tensor data corresponds to the dictionary class in which any dictionary consists of the Kronecker product [10] of smaller subdictionaries [11, 12, 13, 14, 15, 16]. The resulting DL techniques in this instance are interchangeably referred to in the literature as separable DL or Kronecker-structured DL (KS-DL).
In terms of parameters to be estimated and stored, the advantages of KS-DL for tensor data are straightforward: defining and , unstructured dictionary learning uses parameters, whereas the KS-DL model uses only the sum of the subdictionary sizes . Nonetheless, while existing KS-DL methods enjoy lower sample/computational complexity and better storage efficiency over unstructured DL [16], the KS-DL model makes a strong separability assumption among different modes of tensor data. Such an assumption can be overly restrictive for many classes of data [17], resulting in an unfavorable tradeoff between model compactness and representation power.
In this paper, we overcome this limitation by proposing and analyzing a generalization of KS-DL that we interchangeably refer to as learning a mixture of separable dictionaries or low separation rank DL (LSR-DL). The separation rank of a matrix is defined as the minimum number of KS matrices whose sum equals [18, 19]. The LSR-DL model interpolates between the under-parameterized separable model (a special case of LSR-DL model with separation rank ) and the over-parameterized unstructured model. In particular, this model is a natural and consistent way to increase the number of parameters in structured DL (and therefore representation performance) while mimicking the compactness of the KS-DL model. Our numerical experiments confirm the advantages of the LSR-DL model (and algorithms) over both unstructured DL and KS-DL in terms of sample complexity/performance and the amount of memory needed to store the dictionary. Figure 1 also illustrates the difference between LSR-DL and KS-DL: while KS-DL learns dictionary atoms that cannot reconstruct diagonal structures perfectly because of the abundance of axis-aligned (horizontal/vertical) structures within them, LSR-DL also returns dictionary atoms with pronounced diagonal structures as the separation rank increases.
I-A Main Contributions
We propose a new generalization of the separable DL model—which we call a mixture of separable dictionaries model or LSR-DL model—that has a smaller number of parameters than standard DL. We provide conditions under which a true dictionary is recoverable, up to a prescribed error, from tensor-valued training data generated from the LSR-DL model. Our analysis uses the conventional optimization-based formulation of the DL problem [4], except that the search space is constrained to the class of dictionaries with maximum separation rank (and individual mixture terms having bounded norms when and ).111We also provide asymptotic identifiability results for LSR dictionaries without requiring the boundedness assumption; see Section III for details. Due to our choice of Frobenius norm as the distance metric, similar to conventional DL problems, this LSR-DL problem is nonconvex with multiple global minima. Obtaining global convergence guarantees for this highly nonconvex problem is not only difficult but is also insufficient to guarantee global identifiability due to existence of multiple global minima. We therefore focus on local identifiability guarantees, meaning that a search algorithm initialized close enough to the true dictionary can recover that dictionary. Our local identifiability results show that the LSR-DL problem is well posed—i.e., it can return a good estimate of the true dictionary, up to a certain initialization distance, as a solution—and characterize the effect of the separation rank on the sample complexity of the learning problem. To this end, under certain assumptions on the generative model, we show that \Omega\big{(}r(\sum_{n=1}^{N}m_{n}p_{n})p^{2}\rho^{-2}\big{)} samples ensure existence of a local minimum of the constrained LSR-DL problem for th-order tensor data within a neighborhood of radius around the true LSR dictionary.
Our initial local identifiability results are based on an analysis of a separation rank-constrained optimization problem that exploits a connection between LSR (resp., KS) matrices and low-rank (resp., rank-1) tensors. The main challenge that we face as part of this analysis is understanding the topological properties of the class of dictionaries with separation rank at most in terms of compactness and covering number. The resulting insights may be of independent interest to readers for other problems involving LSR matrices.
Next, we note that a result in tensor recovery literature [20] implies finding the separation rank of a matrix is NP-hard. While this means that the rank-constrained LSR-DL problem is computationally intractable, our analysis of this problem provides the basis for our second main contribution, which is development and analysis of two different relaxations of the LSR-DL problem that are computationally tractable in the sense that they do not require explicit computation of the separation rank. The first formulation once again exploits the connection between LSR matrices and low-rank tensors and uses a convex regularizer to implicitly constrain the separation rank of the learned dictionary. The second formulation enforces the LSR structure on the dictionary by explicitly writing it as a summation of KS matrices. Our analyses of the two relaxations once again involve conditions under which the true LSR dictionary is locally recoverable from training tensor data. Our strongest result is for the factorized formulation, described formally in Section II, in which case we derive a sample complexity result that is similar to that of the intractable formulation by finding a correspondence between the local minima of the factorized problem and those of the intractable one. We also compare and contrast the three sets of identifiability results for LSR dictionaries in the body.
In addition to showing the well-posedness of the LSR-DL problem, our theoretical results on local identifiability confirm the advantages of exploiting low-rank structure in tensor problems. Moreover, in order to obtain our results, we acquire a better understanding of the topological properties of the space of LSR matrices. These properties may be useful in other works involving KS and LSR models.
Our third main contribution is the development of practical computational algorithms, which are based on the two relaxations of LSR-DL, for learning of an LSR dictionary in both batch and online settings. We use these algorithms for learning of LSR dictionaries for both synthetic and real tensor data and show their effectiveness in denoising and representation learning tasks. Numerical results obtained as part of these efforts help validate the usefulness of our proposed LSR-DL model and highlight the different strengths and weaknesses of the two LSR-DL relaxations and the corresponding algorithms. In particular, we show empirically that our algorithms provide better representations with a smaller number of parameters than the conventional approach.
I-B Relation to Prior Work
Tensor decompositions [21, 22] are an important tool for avoiding overparameterization of tensor data models in a variety of areas. These include deep learning, collaborative filtering, multilinear subspace learning, source separation, topic modeling, and many other works (see recent surveys [23, 24] and references therein). However, the use of tensor decompositions for reducing the (model and sample) complexity of dictionaries for tensor data has been addressed only recently.
Many recent works provide theoretical analysis for the sample complexity of the conventional DL problem [25, 26, 27, 28]. Among these, Gribonval et al. [27] focus on the local identifiability of the true dictionary underlying vectorized data using Frobenius norm as the distance metric. Shakeri et al. [16] extended this analysis for the sample complexity of the KS-DL problem for th-order tensor data. This analysis relies on expanding the objective function in terms of subdictionaries and exploiting the coordinate-wise Lipschitz continuity property of the objective function with respect to each subdictionary [16]. While this approach ensures the identifiability of the subdictionaries, it requires the dictionary coefficient vectors to follow the so-called separable sparsity model [29] and does not extend to the LSR-DL problem. By contrast, we provide local identifiability sample complexity results for the LSR-DL problem and its two relaxations. Further, our identifiability results hold for coefficient vectors following both the random and separable sparsity models.
In terms of computational algorithms, several works have proposed methods for learning KS dictionaries that rely on alternating minimization techniques to update the subdictionaries [29, 14, 12]. Among other works, Hawe et al. [11] employ a Riemannian conjugate gradient method combined with a nonmonotone line search for KS-DL. While they present the algorithm only for matrix data, its extension to higher-order tensor data is trivial. Schwab et al. [30] have also recently addressed the separable DL problem for matrix data; their contributions include a computational algorithm and global recovery guarantees. In terms of algorithms for LSR-DL, Dantas et al. [13] proposed one of the first methods for matrix data that uses a convex regularizer to impose LSR on the dictionary. One of our batch algorithms, named STARK [1], also uses a convex regularizer for imposing LSR structure. In contrast to Dantas et al. [13], however, STARK can be used to learn a dictionary from tensor data of any order. The other batch algorithm we propose, named TeFDiL, learns subdictionaries of the LSR dictionary by exploiting the connection to tensor recovery and using tensor CPD. Recently, Dantas et al. [31] proposed an algorithm for learning an LSR dictionary for tensor data in which the dictionary update stage is a projected gradient descent algorithm that involves a CPD after every gradient step. In contrast, TeFDiL only requires a single CPD at the end of each dictionary update stage. Finally, while there exist a number of online algorithms for DL [6, 32, 33], the online algorithm developed in here is the first one that enables learning of structured (either KS or LSR) dictionaries.
I-C Organization
In Section II, we provide the necessary background on dictionary learning, introduce our LSR-DL model, and formulate three variants of the LSR-DL problem. In Section III, we show that LSR dictionaries are identifiable using the rank-constrained formulation of the LSR-DL problem. In Section IV, we study the local identifiability of the other two (regularized and factorized) formulations in both asymptotic and finite sample regimes. In Section V, we use the regularized and factorized formulations to design batch and online LSR-DL algorithms, which we evaluate experimentally in Section VI. We conclude the paper and discuss possible future work in Section VII. Proof of Lemma 1 (the rearrangement procedure) is explained in detail in Appendix A. Proofs of technical Lemmas 2, 3, 6, 7, and 8 are provided in Appendix B. A discussion on the convergence of our algorithms is provided in Appendix C.
II Preliminaries and Problem Statement
Notation and Definitions: We use underlined bold upper-case (), bold upper-case (), bold lower-case (), and lower-case () letters to denote tensors, matrices, vectors, and scalars, respectively. For any integer , we define . We denote the -th column of a matrix by . For an matrix and an index set , we denote the matrix constructed from the columns of indexed by as . We denote by an -tuple , while represents the set . We drop the range indicators if they are clear from the context.
Norms and inner products: We denote by the norm of vector (we abuse the terminology in case of ), while we use , , and to denote the spectral, Frobenius, and trace (nuclear) norms of matrix , respectively. Moreover, is the max column norm and . We define the inner product of two tensors (or matrices) and as where is the vectorization operator. We define the Frobenius norm of tensor as . The Euclidean distance between two tuples of the same size is defined as \big{\|}(\mathbf{A}_{n})_{n=1}^{N}-(\mathbf{B}_{n})_{n=1}^{N}\big{\|}_{F}\triangleq\sqrt{\sum_{n=1}^{N}||\mathbf{A}_{n}-\mathbf{B}_{n}||_{F}^{2}}.
Kronecker product: We denote by the Kronecker product of matrices and . We use for the Kronecker product of matrices. We drop the range indicators when there is no ambiguity. We call a matrix a (-th order) Kronecker-structured (KS) matrix if it is a Kronecker product of matrices.
Definitions for matrices: For a matrix with unit -norm columns, we define the cumulative coherence as . We say a matrix satisfies the -restricted isometry property (-RIP) with constant if for any and any with , we have .
Definitions for tensors: We briefly present required tensor definitions here: see Kolda and Bader [21] for more details. The mode- unfolding matrix of is denoted by , where each column of consists of the vector formed by fixing all indices of except the one in the th-order. We denote the outer product (tensor product) of vectors by , while denotes the mode- product between a tensor and a matrix. An -way tensor is rank- if it can be written as outer product of vectors: . Throughout this paper, by the rank of a tensor, , we mean the CP-rank of , the minimum number of rank- tensors that construct as their sum. The CP decomposition (CPD), decomposes a tensor into sum of its rank- tensor components. The Tucker decomposition factorizes an -way tensor as , where denotes the core tensor and denote factor matrices along the -th mode of for .
Notations for functions and spaces: We denote the element-wise sign function by . For any function , we define the difference . We denote by the Euclidean unit sphere: . We also denote the Euclidean sphere with radius by . The oblique manifold in is the manifold of matrices with unit-norm columns: . We drop the dimension subscripts and use only when there is no ambiguity. The covering number of a set with respect to a norm , denoted by , is the minimum number of balls of -norm radius needed to cover .
Dictionary Learning Setup: In dictionary learning (DL) for vector data, we assume observations are generated according to the following model:
[TABLE]
where is the true underlying dictionary, is a randomly generated sparse coefficient vector, and is the observation noise vector. The goal in DL is to recover the true dictionary given the noisy observations that are independent realizations of (1). The ideal objective is to solve the statistical risk minimization problem
[TABLE]
where is the underlying distribution of the observations, is the dictionary class, typically selected for vector data to be the same as the oblique manifold, and
[TABLE]
However, since we have access to the distribution only through noisy observations drawn from this distribution, we resort to solving the following empirical risk minimization problem as a proxy for Problem (2):
[TABLE]
Dictionary Learning for Tensor Data: To represent tensor data, conventional DL approaches vectorize tensor data samples and treat them as one-dimensional arrays. One way to explicitly account for the tensor structure in data is to use the Kronecker-structured DL (KS-DL) model, which is based on the Tucker decomposition of tensor data. In the KS-DL model, we assume that observations are generated according to
[TABLE]
where are generating subdictionaries, and and are the coefficient and noise tensors, respectively. Equivalently, the generating model (5) can be stated for as:
[TABLE]
where and [21]. This is the same as the unstructured model with the additional condition that the generating dictionary is a Kronecker product of subdictionaries. As a result, in the KS-DL problem, the constraint set in (4) becomes , where is the set of KS matrices with unit-norm columns and and are vectors containing ’s and ’s, respectively.222We have changed the indexing of subdictionaries for ease of notation.
In summary, the structure in tensor data is exploited in the KS-DL model by assuming the dictionary is “separable” into subdictionaries for each mode. However, as discussed earlier, this separable model is rather restrictive. Instead, we generalize the KS-DL model using the notion of separation rank.333The term was introduced in Tsiligkaridis and Hero [19] for (see also Beylkin and Mohlenkamp [18]).
Definition 1**.**
The separation rank of a matrix is the minimum number of th-order KS matrices such that , where .
The KS-DL model corresponds to dictionaries with separation rank . We instead propose the low separation rank (LSR) DL model in which the separation rank of the underlying dictionary is relatively small so that . This generalizes the KS-DL model to a generating dictionary of the form , where is the separation rank of . Consequently, defining , the empirical rank-constrained LSR-DL problem is
[TABLE]
However, the analytical tools at our disposal require the constraint set in (7) to be closed, which we show does not hold for when and . In that case, we instead analyze (7) with replaced by () closure of and () a certain closed subset of . We refer the reader to Section III for further discussion.
In our study of the LSR-DL model (which includes the KS-DL model as a special case), we use a correspondence between KS matrices and rank-1 tensors, stated in Lemma 1 below, which allows us to leverage techniques and results in the tensor recovery literature to analyze the LSR-DL problem and develop tractable algorithms. (This correspondence was first exploited in our earlier work [1].)
Lemma 1**.**
Any th-order Kronecker-structured matrix can be rearranged as a rank-, th-order tensor with .
Figure 2 provides an example of the rearrangement procedure, which involves finding corresponding indices on the KS matrix and the tensor. A proof of Lemma 1, which includes details of the rearrangement strategy, is provided in Appendix A. It follows immediately from Lemma 1 that if , then we can rearrange matrix into the tensor where . Hence, we have the following equivalence:
[TABLE]
This correspondence between separation rank and tensor rank highlights a challenge with the LSR-DL problem: finding the rank of a tensor is NP-hard[20] and thus so is finding the separation rank of a matrix. This makes Problem (7) in its current form (and its variants) intractable. To overcome this limitation, we introduce two tractable relaxations to the rank-constrained Problem (7) that do not require explicit computation of the tensor rank. The first relaxation uses a convex regularization term to implicitly impose low tensor rank structure on , which results in a low separation rank . The resulting empirical regularization-based LSR-DL problem is
[TABLE]
with , where is described in (3) and is a convex regularizer to enforce low-rank structure on . The second relaxation is a factorization-based LSR-DL formulation in which the LSR dictionary is explicitly written in terms of its subdictionaries. The resulting empirical risk minimization problem is
[TABLE]
where with
[TABLE]
and the terms are constrained as for some positive constant when and .
In the rest of this paper, we study the problem of identifying the true underlying LSR-DL dictionary by analyzing the LSR-DL Problems (7)–(9) introduced in this section and developing algorithms to solve Problems (8) and (9) in both batch and online settings. Note that while Problem (7) (and its variants when and ) cannot be explicitly solved because of its NP-hardness, identifiability analysis of this problem—provided in Section III—provides the basis for the analysis of tractable Problems (8) and (9), provided in Section IV. To improve the readability of our notation-heavy discussions and analysis, we have provided a table of notations (Table I) for easy access to definitions of the most commonly used notation.
III Identifiability in the Rank-constrained LSR-DL Problem
In this section, we derive conditions under which a dictionary is identifiable as a solution to either the separation rank-constrained problem in (7) or a slight variant of (7) when and . Specifically, we show that under certain assumptions on the generative model, there is at least one local minimum of either Problem (7) or one of its variants that is “close” to the underlying dictionary . Notwithstanding the fact that no efficient algorithm exists to solve the intractable Problem (7), this identifiability result is important in that it lays the foundation for the local identifiability results in tractable Problems (8) and (9).
Generative Model: Let be the underlying dictionary. Each tensor data sample in its vectorized form is independently generated using a linear combination of atoms of dictionary with added noise: , where . Specifically, atoms of are selected uniformly at random, defining the support . Then, we draw a random sparse coefficient vector supported on . We state further assumptions on our model similar to prior works [27, 16].
Assumption 1** (Coefficient Distribution).**
Consider a random variable and positive constants and . Define . We assume: i) \mathbb{E}\big{\{}\mathbf{x}_{\mathcal{J}}^{0}[\mathbf{x}_{\mathcal{J}}^{0}]^{T}|\mathcal{J}\big{\}}=\mathbb{E}\{x^{2}\}\cdot\mathbf{I}_{s}, ii) \mathbb{E}\big{\{}\mathbf{s}_{\mathcal{J}}^{0}[\mathbf{s}_{\mathcal{J}}^{0}]^{T}|\mathcal{J}\big{\}}=\mathbf{I}_{s}, iii) \mathbb{E}\big{\{}\mathbf{s}_{\mathcal{J}}^{0}[\mathbf{x}_{\mathcal{J}}^{0}]^{T}|\mathcal{J}\big{\}}=\mathbb{E}\{|x|\}\cdot\mathbf{I}_{s}, and iv) and almost surely.
Assumption 2** (Noise Distribution).**
Consider a random variable and positive constant . We assume: i) \mathbb{E}\big{\{}\bm{\epsilon}\bm{\epsilon}^{T}|\mathcal{J}\big{\}}=\mathbb{E}\{\epsilon^{2}\}\cdot\mathbf{I}_{m}, ii) \mathbb{E}\big{\{}\mathbf{x}^{0}\bm{\epsilon}^{T}|\mathcal{J}\big{\}}=\mathbb{E}\big{\{}\mathbf{s}^{0}\bm{\epsilon}^{T}|\mathcal{J}\big{\}}=0, and iii) almost surely.
Note that Assumptions 1-iv and 2-iii imply the magnitude of is bounded: . Next, we define positive parameters , , and for ease of notation. We use the following assumption, similar to Gribonval et al. [27, Thm. 1].
Assumption 3**.**
Assume , , , , and the noise is relatively small in the sense that .
Our Approach: In our analysis of the separation rank-constrained LSR-DL problem, we will alternate between four different constraint sets that are related to our dictionary class , namely, , , the closure of under the Frobenius norm, and a closed subset of , defined as . We often use the generic notation for the constraint set when our discussion is applicable to more than one of these sets.
We want to find conditions that imply the existence of a local minimum of within a ball of radius around the true dictionary :
[TABLE]
for some small . To this end, we first show that the expected risk function in (2) has a local minimum in for the LSR-DL constraint set .
To show that a local minimum of exists in , we need to show that attains its minimum over in the interior of .444 Having a minimum on the boundary is not sufficient. If the minimizer of over is on the boundary of , the value of in the neighborhood of outside can be smaller than ; therefore, is not necessarily a local minimum of . We show this in two stages. First, we use the Weierstrass Extreme Value Theorem [34], which dictates that the continuous function attains a minimum in (or on the boundary of) as long as is a compact set. Therefore, we first investigate compactness of in Section III-A. Second, in order to be certain that the minimizer of over is a local minimum of , we show that cannot obtain its minimum over on the boundary of , denoted by . To this end, in Section III-B we derive conditions that if is nonempty then we have555If the boundary is empty, it is trivial that the infimum is attained in the interior of the set.
[TABLE]
which implies cannot achieve its minimum on .
Finally, in Section III-C we use concentration of measure inequalities to relate in (4) to and find the number of samples needed to guarantee (with high probability) that also has a local minimum in the interior of .
III-A Compactness of the Constraint Sets
When the constraint set is a compact subset of the Euclidean space , the subset is also compact. Thus, we first investigate the compactness of the constraint set . Since is a bounded set, according to the Heine-Borel Theorem [34], it is a compact subset of if and only if it is closed. Also, can be written as the intersection of and the oblique manifold . In order for to be closed, it suffices to show that and are closed. It is trivial to show is closed; hence, we focus on whether is closed.
In the following, we use the facts that the constraint is equivalent to and that the rearrangement mapping that sends to preserves topological properties of sets such as the distances between the set elements under the Frobenius norm. These facts allow us to translate the topological properties of tensor sets into properties of the structured matrices that we study here.
Remark. Proofs of Lemmas 2, 3, 6, 7, and 8 are provided in Appendix B.
Lemma 2**.**
Let and . Then, the set is not closed. However, the set of KS matrices and the set are closed.
To illustrate the non-closedness of for and and motivate the use of the sets and in lieu of , we provide an example.
Example. Consider the sequence where are linearly independent pairs. Here, for any . The limit point of this sequence is , which is a separation-rank- matrix. Hence, the set is not closed.
The non-closedness of means there exist sequences in whose limit points are not in the set. Two possible solutions to circumvent this issue include: () use the closure of as the constraint set, and () eliminate such sequences from . We discuss each solution in detail below.
Adding the limit points
We denote the closure of by . By slightly relaxing the constraint set in (7) to , we can instead solve the following:
[TABLE]
where . Note that () a solution to (7) is a solution to (12) and () a solution to (12) is either a solution to (7) or is arbitrarily close to a member of .666The first argument holds since if for all , by continuity it also holds for all . The second argument is trivial.
Eliminating the problematic sequences
In order to exclude the sequences such that for all and , we first need to characterize them.
Lemma 3**.**
Assume where and . We can write where . Then, as . In fact, at least two of the coefficient sequences are unbounded.
The following corollary of Lemma 3 suggests that one can exclude the problematic sequences from by bounding the norm of individual KS (separation-rank-) terms.
Corollary 1**.**
*Consider the set whose members can be written as such that . Then, for any the set {}^{c}{\mathcal{L}}^{N,r}_{\mathbf{m},\mathbf{p}}=\big{\{}\mathbf{D}\in\mathcal{L}^{N,r}_{\mathbf{m},\mathbf{p}}|\big{\|}\bigotimes\mathbf{D}^{k}_{n}\big{\|}_{F}\leq c\big{\}} is closed. *
We have now shown that the sets , , , and are compact subsets of .
III-B Asymptotic Analysis for Dictionary Identifiability
Now that we have discussed the compactness of the relevant constraint sets, we are ready to show that the minimum of over , defined in (10), is not attained on . This will complete our proof of existence of a local minimum of in . In our proof, we make use of a result in Gribonval et al. [27], presented here in Lemma 4.
Lemma 4** (Theorem 1 in Gribonval et al. [27]).**
Consider the statistical DL Problem (2) with constraint set . Suppose the generating dictionary and Assumptions 1–3 hold. Then, for any such that and , we have
[TABLE]
for all such that .
Interested readers can find the detailed proof of Lemma 4 in Gribonval et al. [27]. The following theorem states our first identifiability result for the LSR-DL model.
Theorem 1**.**
Consider the statistical DL Problem (2) with constraint set being either , , or . Suppose the generating dictionary and Assumptions 1–3 hold. Then, for any such that and , the function has a local minimum such that .
Proof.
Since is a continuous function and the ball is compact, the function attains its infimum at a point in the ball. If this minimum is attained in the interior of then it is a local minimum of . Therefore, a key ingredient of the proof is showing that for all if is nonempty. Lemma 4 states the conditions under which on , where {\mathcal{S}}_{\rho}\triangleq\big{\{}\mathbf{D}\in\mathcal{D}~{}\big{|}~{}\left\|\mathbf{D}-\mathbf{D}^{0}\right\|_{F}\leq\rho\big{\}}.
Since , the result of Lemma 4 can be used for our problem as well, i.e. for any , we have , when . It follows from this result together with the existence of the infimum of in that Problem (2) has a local minimum within a ball of radius around the true dictionary . ∎
In Theorem 1, we guarantee that the true dictionary, , is identifiable as a solution to the statistical rank-constrained LSR-DL problem (2). Next, we take advantage of concentration of measure inequalities that relate the empirical objective in (4) to the statistical objective studied in Theorem 1 to find the number of samples needed to ensure is also identifiable via the empirical rank-constrained LSR-DL problem (2).
III-C Sample Complexity for Dictionary Identifiability
We now derive the number of samples required to guarantee, with high probability, that has a local minimum at a point “close” to when the constraint set is either , , or for and . First, we use concentration of measure inequalities based on the covering number of the dictionary class to show that the empirical loss uniformly converges to its expectation with high probability. This is formalized below.
Lemma 5** (Theorem 1 and Lemma 11, Gribonval et al. [35]).**
Consider the empirical DL Problem (4) and suppose Assumptions 1 and 2 are satisfied. For any and constants and , with probability at least we have
[TABLE]
where is such that .
Define . It follows from (14) that with high probability (w.h.p.),
[TABLE]
for all . Therefore, when for all , we have for all . In this case, we can use similar arguments as in the asymptotic analysis to show that has a local minimum at a point in the interior of . Hence, our focus in this section is on finding the sample complexity required to guarantee that w.h.p. We begin with characterization of covering numbers of the three constraint sets, which may also be of independent interest to some readers.
Covering Numbers: The covering number of the set with respect to the norm is known in the literature to be upper bounded as follows [35]:
[TABLE]
We now turn to finding the covering numbers of LSR sets and . The following lemma establishes a bound on covering number of , which depends on the separation rank exponentially.
Lemma 6**.**
The covering number of the set with respect to the norm is upper bounded as follows:
[TABLE]
Next, we obtain an upper bound on the covering number of for a given constant .
Lemma 7**.**
The covering number of the set with respect to the max-column norm is bounded as follows:
[TABLE]
We can now find the sample complexity of the LSR-DL Problem (4) by plugging in the values of and in Lemma 5.
Theorem 2**.**
Consider the empirical LSR dictionary learning Problem (4) with constraint set being , , or . Fix any . Suppose the generating dictionary and Assumptions 1–3 are satisfied. Assume and . Define a constant that depends on the dictionary class:
- •
* and when ,*
- •
* and when ,*
- •
* and when .*
Then, given a number of samples satisfying
[TABLE]
where is a constant, with probability no less than , the empirical risk objective function has a local minimizer such that .
Proof.
We take a similar approach to the proof of Theorem 1. Due to compactness of the ball and continuity of , it follows that attains its minimum at a point in . It remains to show that for all which implies existence of a local minimizer of at such that .
Inequality (15) shows that it suffices to set to have . From Lemma 5 we know . Therefore, using the lower bound (13) on we have with probability at least
[TABLE]
with and 777Under the conditions of this theorem, , where denotes the RIP constant of .. Rearranging, we get
[TABLE]
Setting and we get the lower bound
[TABLE]
with probability at least . Given that the number of samples satisfies (17) for , with high probability for any . Therefore, it follows from the existence of the infimum of in that has a local minimum at a point within a ball of radius around the true dictionary . ∎
The \Omega\big{(}r(\sum_{n}m_{n}p_{n})p^{2}\rho^{-2}\big{)} sample complexity upper bound we obtain here for rank-constrained LSR-DL is a reduction compared to the sample complexity of standard DL [27]. However, the minimax lower bound scaling of for KS-DL [15] () indicates an gap with our sample complexity upper bound. This gap could be due to looseness in the lower bound, our upper bound, or both. We leave an investigation of this and possible tightening of the bound(s) to future work.
IV Identifiability in the Tractable LSR-DL Problems
In Section II, we introduced two tractable relaxations to the rank-constrained LSR-DL problem: a regularized problem (8) with a convex regularization term and a factorization-based problem (9) in which the dictionary is written in terms of its subdictionaries. We now provide results on the local identifiability of the true dictionary in these problems, i.e., we find conditions under which at least one local minimizer of these problems is located near the true dictionary . Such local identifiability result implies that any DL algorithm that converges to a local minimum of these problems can recover up to a small error if it is initialized close enough to .
IV-A Regularization-based LSR Dictionary Learning
The first tractable LSR-DL problem that we study is the regularized problem (8). Exploiting the relation between and , the LSR structure is enforced on the dictionary by a convex regularizer that imposes low tensor rank structure on . The regularizer that we use here is a commonly used convex proxy for the tensor rank function, the sum-trace-norm [36], which is defined as the average of the trace (nuclear) norms of the unfoldings of the tensor: .
The first question we address is whether the reference dictionary that generates the observations is identifiable via Problem (8). Our local identifiability result here is limited to when , i.e. the true dictionary is KS. For such , we show that there is at least one local minimizer of under Assumptions 1–3 that is close to .
Theorem 3**.**
Consider the regularized LSR-DL problem (8). Suppose that the generating dictionary and Assumptions 1–3 are satisfied. Moreover, let and . Then, the expected risk function has a local minimizer such that .
Moreover, given samples such that
[TABLE]
where and are positive constants, then, we have with probability no less than that the empirical risk function has a local minimum at such that .
Proof.
Consider the ball . Compactness of and continuity of guarantee that attains its minimum at a point in . Similarly, reaches its minimum at a point in . We now need to show in either case the minimum is not attained on the boundary of . To this end, we show in the following that and for any .
Incorporation of the trace-norm regularization term in (8) within the objective in (4) introduces a factor to and . We know from Lemma 1 that when the true dictionary is a KS matrix (), its rearrangement tensor is a rank- tensor and therefore all unfoldings of are rank- matrices. This implies . Moreover, for all we have \big{\|}\mathbf{D}^{(n)}\big{\|}_{F}=\big{\|}[\mathbf{D}^{0}]^{(n)}\big{\|}_{F}=\sqrt{p}. Therefore,
[TABLE]
where and denotes the -th singular value of . Therefore, we conclude that and for any . According to Lemma 4, for all on the boundary of the ball . Furthermore, under the assumptions of the current theorem, given a number of samples satisfying (19), Gribonval et al. [27] show that the empirical difference for all on the boundary of {\mathcal{S}}_{\rho}=\big{\{}\mathbf{D}\in\mathcal{D}~{}\big{|}~{}\left\|\mathbf{D}-\mathbf{D}^{0}\right\|_{F}\leq\rho\big{\}}, and therefore on the boundary of , with probability at least . Therefore, for both and , the minimum is attained in the interior of and not on its boundary. ∎
We discuss the implications of Theorem 3 in Section IV-C.
IV-B Factorization-based LSR Dictionary Learning
We now shift our focus to Problem (9), which expands as and optimizes over the individual subdictionaries, and show that there is at least one local minimum of the factorization-based LSR-DL Problem (9) such that is close to the underlying dictionary . Our strategy here is to establish a connection between the local minima of (9) and those of (4). Specifically, we show that when the dictionary class in (7) matches that of (9), for every local minimum of (4), there exists a local minimum of (9) such that . Furthermore, we use the result of Theorems 1 and 2 that there exists a local minimum of Problem (4) within a small ball around . It follows from these facts that under the generating model considered here, a local minimum of (9) is such that is close to .
We begin with a bound on the distance between LSR matrices when the tuples of their factor matrices are -close.
Lemma 8**.**
For any two tuples and such that for all and , if the distance \big{\|}(\mathbf{A}^{k}_{n})-(\mathbf{B}^{k}_{n})\big{\|}_{F}\leq\epsilon then \big{\|}\sum\nolimits_{k=1}^{r}\bigotimes\mathbf{A}^{k}_{n}-\sum_{k=1}^{r}\bigotimes\mathbf{B}^{k}_{n}\big{\|}_{F}\leq\alpha^{N-1}\sqrt{Nr}\epsilon.
Theorem 4**.**
Consider the factorization-based LSR-DL problem (9). Suppose that Assumptions 1–3 are satisfied and with . Then, the expected risk function \mathbb{E}[F^{\mathrm{fac}}_{\mathbf{Y}}\big{(}\{\mathbf{D}^{k}_{n}\}\big{)}] has a local minimizer such that .
Moreover, when the sample complexity requirements (17) are satisfied for some positive constant , then with probability no less than the empirical risk objective function F^{\mathrm{fac}}_{\mathbf{Y}}\big{(}\{\mathbf{D}^{k}_{n}\}\big{)} has a local minimum achieved at such that .
Proof.
Let us first consider the finite sample case. Theorem 2 shows existence of a local minimizer of Problem (7) for constraint sets , , and , such that w.h.p. Here, we want to show that for such , there exists a such that and is a local minimizer of Problem (9).
First, let us consider Problem (7) with . It is easy to show that any can be written as such that for all and , without loss of generality, where . Define \mathcal{C}^{\mathrm{fac}}\triangleq\left\{(\mathbf{D}^{k}_{n})\big{|}\sum\bigotimes\mathbf{D}^{k}_{n}\in{}^{c}{\mathcal{K}}^{N,r}_{\mathbf{m},\mathbf{p}}:\forall k,n,\mathbf{D}^{k}_{n}\in\alpha\mathcal{\mathcal{U}}_{m\times p}\right\}. Since , there is a (with for all and ) such that . According to Lemma 8, for any it follows from \big{\|}(\mathbf{D}^{k}_{n})-([\mathbf{D}^{k}_{n}]^{*})\big{\|}_{F}\leq\epsilon^{\prime} that \big{\|}\sum\bigotimes\mathbf{D}^{k}_{n}-\sum\bigotimes[\mathbf{D}^{k}_{n}]^{*}\big{\|}_{F}\leq\alpha^{N-1}\sqrt{Nr}\epsilon^{\prime}. Since is a local minimizer of (7), there exists a positive such that for all satisfying , we have . If we choose small enough such that and , then for any such that and \big{\|}(\mathbf{D}^{k}_{n})-([\mathbf{D}^{k}_{n}]^{*})\big{\|}_{F}\leq\epsilon^{\prime}, we have \big{\|}\sum\bigotimes\mathbf{D}^{k}_{n}-\mathbf{D}^{*}\big{\|}_{F}\leq\epsilon and this means that F^{\mathrm{fac}}_{\mathbf{Y}}\big{(}\{\mathbf{D}^{k}_{n}\}\big{)}-F^{\mathrm{fac}}_{\mathbf{Y}}\big{(}\{[\mathbf{D}^{k}_{n}]^{*}\}\big{)}=F_{\mathbf{Y}}(\sum\bigotimes\mathbf{D}^{k}_{n})-F_{\mathbf{Y}}(\mathbf{D}^{*})\geq 0. Therefore, is a local minimizer of Problem (9). This concludes our proof for the finite sample case with constraint set .
Note that we can write and with . Therefore, the above results also hold for and since they are special cases of .
It is easy to see similar relation exists between the local minima of and , proving the asymptotic result in the statement of this theorem. ∎
IV-C Discussion
In this section, we discuss the local identifiability of the true dictionary in the regularization-based formulation (Theorem 3) and the factorization-based formulation (Theorem 4). For the regularization-based formulation, our results only hold for the case where the true dictionary is KS, i.e. . We obtain sample complexity requirement of in this case, which matches the sample complexity requirement of the unstructured formulation [27]. This is due to the fact that the class of dictionaries in the regularization-based formulation is , i.e., the LSR constraint is not explicitly imposed in this formulation. Thus, our covering number-based approach does not provide improved sample complexity results compared to the unstructured formulation. The experimental results of the regularization-based algorithm STARK (see Section VI) suggest there is room to improve our sample complexity result for the regularized formulation. We leave investigating such improvement as future work. Nonetheless, our results imply well-posedness of the regularized LSR-DL problem.
For the factorization-based formulation, we show that samples are required for local identifiability of a dictionary of separation-rank . This result matches that of our rank-constrained formulation stated in Theorem 2. Note that when the separation rank is , this result gives a bound on the sample complexity of the KS-DL model as a special case. To illustrate the implication of our bound, consider the case of and . In this case, our bound scales as , which results in reduction in sample complexity scaling compared to the unstructured DL bound [27]. In the case of and (unstructured DL), our bound scales as , which is consistent with the unstructured DL bound [27]. Note that unlike the KS-DL analysis [16], which shows a necessary sample complexity of , our analysis of the factorized model does not ensure identifiability of the true subdictionaries in the LSR-DL model. However, the results for KS-DL require the dictionary coefficient vectors to follow the separable sparsity model. In contrast, our result does not require any constraints on the sparsity pattern of coefficients.
V Computational Algorithms
In Section IV, we showed that the tractable LSR-DL Problems (8) and (9) each have at least one local minimum close to the true dictionary. In this section we develop algorithms to find these local minima. Solving Problems (8) and (9) require minimization with respect to (w.r.t.) . Therefore, similar to conventional DL algorithms, we introduce alternating minimization-type algorithms that at every iteration, first perform minimization of the objective function w.r.t. (sparse coding stage) and then minimize the objective w.r.t. the dictionary (dictionary update stage).
The sparse coding stage is a simple Lasso problem and remains the same in our algorithms. However, the algorithms differ in their dictionary update stages, which we discuss next.
Remark. We leave the formal convergence results of our algorithms to future work. However, we provide a discussion on challenges and possible approaches to establish convergence of our algorithms in Appendix C.
V-A STARK: A Regularization-based LSR-DL Algorithm
We first discuss an algorithm, which we term STructured dictionAry learning via Regularized low-ranK Tensor Recovery (STARK), that helps solve the regularized LSR-DL problem given in (8) and discussed in Section IV using the Alternating Direction Method of Multipliers (ADMM) [37].
The main novelty in solving (8) using is the dictionary update stage. This stage, which involves updating for a fixed set of sparse codes , is particularly challenging for gradient-based methods because the dictionary update involves interdependent nuclear norms of different unfoldings of the rearranged tensor . Inspired by many works in the literature on low-rank tensor estimation [38, 39, 36], we instead suggest the following reformulation of the dictionary update stage of (8):
[TABLE]
In this formulation, although the nuclear norms depend on one another through the introduced constraint, we can decouple the minimization problem into separate subproblems. To solve this problem, we first find a solution to the problem without the constraint , then project the solution onto by normalizing the columns of . We adopt this approximation to avoid the complexity of solving the problem with the constraint . Such approach has been used in prior works; see, e.g., [13, 29]. We can solve the objective function (V-A) (without constraint) using ADMM, which involves decoupling the problem into independent subproblems by forming the following augmented Lagrangian:
[TABLE]
where is shorthand for . In order to find the gradient of (V-A) with respect to , we rewrite the Lagrangian function in the following form:
[TABLE]
Here, (not to be confused with our earlier use of for )) and the linear operator , where and is a permutation matrix such that . The procedure to find is explained in Appendix A.
ADMM Update Rules: Each iteration of ADMM consists of the following steps [37]:
[TABLE]
for all . The solution to (22) can be obtained by taking the gradient of w.r.t. and setting it to zero. Suppressing the iteration index for ease of notation, we have
[TABLE]
where \mathcal{T}^{*}(\mathbf{v})=\mathop{\mathrm{vec}}\nolimits^{-1}\big{(}\mathbf{\Pi}\widetilde{\mathbf{X}}\mathbf{v}\big{)} is the adjoint of the linear operator [39]. Setting the gradient to zero results in
[TABLE]
Equivalently, we have
[TABLE]
Therefore, suppressing the index , the update rule for is
[TABLE]
To update , we can further split (23) into independent subproblems (suppressing the index ):
[TABLE]
We can reformulate as
[TABLE]
The minimizer of with respect to is where applies soft thresholding at level on the singular values of matrix [40]. Therefore, suppressing the index ,
[TABLE]
where is the inverse of the unfolding operator. Algorithm 1 summarizes this discussion and provides pseudocode for the dictionary update stage in STARK.
V-B TeFDiL: A Factorization-based LSR-DL Algorithm
While our experiments in Section VI validate good performance of STARK, the algorithm finds the dictionary and not the subdictionaries . Moreover, STARK only allows indirect control over the separation rank of the dictionary through the regularization parameter . This motivates developing a factorization-based LSR-DL algorithm that can find the subdictionaries and allows for direct tuning of the separation rank to control the number of parameters of the model. To this end, we propose a factorization-based LSR-DL algorithm termed Tensor Factorization-Based DL (TeFDiL) in this section for solving Problem (9).
We discussed earlier in Section V-A that the error term can be reformulated as where . Thus, the dictionary update objective in (9) can be reformulated as where . To avoid the complexity of solving this problem, we resort to first obtaining an inexact solution by minimizing over and then enforcing the low-rank structure by finding the rank- approximation of the minimizer of . TeFDiL employs CP decomposition (CPD) to find this approximation and thus enforce LSR structure on the updated dictionary.
Assuming the matrix of sparse codes is full row-rank999In our experiments, we add to with a small at every iteration to ensure full-rankness., then is full column-rank and \underline{\mathbf{A}}=\mathcal{T}^{+}(\mathbf{y})=\mathop{\mathrm{vec}}\nolimits^{-1}\big{(}\mathbf{\Pi}\big{(}\widetilde{\mathbf{X}}\widetilde{\mathbf{X}}^{T}\big{)}^{-1}\widetilde{\mathbf{X}}\mathbf{y}\big{)} minimizes . Now, it remains to solve the following problem to update :
[TABLE]
Although finding the best rank- approximation (-term CPD) of a tensor is ill-defined in general [41], various numerical algorithms exist in the tensor recovery literature to find a “good” rank- approximation of a tensor [21, 41]. TeFDiL can employ any of these algorithms to find the -term CPD, denoted by , of . At the end of each dictionary update stage, the columns of are normalized. Algorithm 2 describes the dictionary update step of TeFDiL.
V-C OSubDil: An Online LSR-DL Algorithm
Both STARK and TeFDiL are batch methods in that they use the entire dataset for DL in every iteration. This makes them less scalable with the size of datasets due to high memory and per iteration computational cost and also makes them unsuitable for streaming data settings. To overcome these limitations, we now propose an online LSR-DL algorithm termed Online SubDictionary Learning for structured DL (OSubDil) that uses only a single data sample (or a small mini-batch) in every iteration (see Algorithm 3). This algorithm has better memory efficiency as it removes the need for storing all data points and has significantly lower per-iteration computational complexity. In OSubDil, once a new sample arrives, its sparse representation is found using the current dictionary estimate and then the dictionary is updated using and . The dictionary update stage objective function after receiving the -th sample is
[TABLE]
We can rewrite this objective as
[TABLE]
where, dropping the iteration index , the matrix and the estimate . We can further define the matrices and . To minimize with respect to each , we take a similar approach as in Mairal et al. [6] and use a (block) coordinate descent algorithm with warm start to update the columns of in a cyclic manner. Algorithm 3 describes the dictionary update step of OSubDil.
VI Numerical Experiments
We evaluate our algorithms on synthetic and real-world datasets to understand the impact of training set size and noise level on the performance of LSR-DL. In particular, we want to understand the effect of exploiting additional structure in representation accuracy and denoising performance. We compare the performance of our proposed algorithms with existing DL algorithms in each scenario and show that in almost every case our proposed LSR-DL algorithms outperform -SVD [5]. Our results also offer insights into how the size and quality of training data can affect the choice of the proper DL model. Specifically, our experiments on image denoising show that when the noise level in data is high, TeFDiL performs best when the separation rank is but in low noise regimes, its performance improves as we increase the separation rank. Furthermore, our synthetic experiments confirm that when the true underlying dictionary follows the KS (LSR) structure, our structured algorithms clearly outperform -SVD, especially when the number of training samples is very small. This implies that our algorithms should perform well in applications where the true dictionary is close to being LSR-structured.
Remark. In all our experiments, hyperparameters and (for STARK), (for SubDil), and regularization parameter for sparsity, , have been selected using cross-validation on each training dataset based on representation error. The only exception is for in cases where we specify its value in the KS-DL experiments () in Table II and in the TeFDiL experiments reported in Table III.
Synthetic Experiments: We compare our algorithms to -SVD (standard DL) as well as a simple block coordinate descent (BCD) algorithm that alternates between updating every subdictionary in problem (9). This BCD algorithm can be interpreted as an extension of the KS-DL algorithm [29] for the LSR model. We show how structured DL algorithms outperform the unstructured algorithm -SVD [5] when the underlying dictionary is structured, especially when the training set is small. We focus on 3rd-order tensor data and we randomly generate a KS dictionary with dimensions and . We select i.i.d samples from the standard Gaussian distribution, , for the subdictionary elements, and then normalize the columns of the subdictionaries. To generate , we select the locations of nonzero elements uniformly at random. The values of those elements are sampled i.i.d. from . We assume observations are generated according to . In the initialization stage of the algorithms, is initialized using random columns of for -SVD and random columns of the unfoldings of for the structured DL algorithms. Sparse coding is performed using OMP[42]. Due to the invariance of DL to column permutations in the dictionary, we choose reconstruction error as the performance criteria. For , -SVD cannot be used since . Reconstruction errors are plotted in Figure 3(a). It can be seen that for small number of samples, TeFDiL outperforms all three algorithms BCD, -SVD, and STARK. As more samples become available, both TeFDiL and STARK (the proposed algorithms) outperform BCD and -SVD.
Real-world Experiments: In this set of experiments, we evaluate the image denoising performance of different DL algorithms on four RGB images, House, Castle, Mushroom, and Lena, which have dimensions , , , and , respectively. We corrupt the images using additive white Gaussian noise with standard deviations . To construct the training data set, we extract overlapping patches of size from each image and treat each patch as a 3-dimensional data sample. We learn dictionaries with parameters and . In the training stage, we perform sparse coding using FISTA [43] (to reduce training time) with regularization parameter for all algorithms. To perform denoising, we use OMP with . To evaluate the denoising performances of the methods, we use the resulting peak signal to noise ratio (PSNR) of the reconstructed images [44]. Table II demonstrates the image denoising results.
LSR-DL vs Unstructured DL: We observe that STARK outperforms -SVD in every case when the noise level is high and in most cases when the noise level is low. Moreover, TeFDiL outperforms -SVD in both low-noise and high-noise regimes for all four images while having considerably fewer parameters (one to three orders of magnitude).101010While the improvements in image denoising reported in DL papers are sometimes below 0.5 dB [11, 5, 45, 29]), we show that our algorithms provide 1–3 dB improvements over -SVD in most scenarios.
LSR-DL vs KS-DL: Our LSR-DL methods outperform SeDiL [11] and while BCD [29] has a good performance for , its denoising performance suffers when noise level increases.111111Note that SeDiL results may be improved by careful parameter tuning.
Table III demonstrates the image denoising performance of TeFDiL for Mushroom based on the separation rank of TeFDiL. When the noise level is low, performance improves with increasing the separation rank. However, for higher noise level , increasing the number of parameters has an inverse effect on the generalization performance.
Comparison of LSR-DL Algorithms: We compare LSR-DL algorithms BCD, STARK and TeFDiL. As for the merits of our LSR-DL algorithms over BCD, our experiments show that both TeFDiL and STARK outperform BCD in both noise regimes. In addition, while TeFDiL and STARK can be easily and efficiently used for higher separation rank dictionaries, when the separation rank is higher, BCD with higher rank does not perform well. While STARK has a better performance than TeFDiL for some tasks, it has the disadvantage that it does not output the subdictionaries and does not allow for direct tuning of the separation rank. Ultimately, the choice between these two algorithms will be application dependent. The flexibility in tuning the number of KS terms in the dictionary in TeFDiL (and indirectly in STARK, through parameter ) allows selection of the number of parameters in accordance with the size and quality of the training data. When the training set is small and noisy, smaller separation rank (perhaps ) results in a better performance. For training sets of larger size and better quality, increasing the separation rank allows for higher capacity to learn more complicated structures, resulting in a better performance.
OSubDil vs Online (Unstructured) DL: Figure 3(b) shows the PSNR for reconstructing House using OSubDil and Online DL in Mairal et al. [6] based on the number of observed samples. We observe that in the presence of high level of noise, our structured algorithm is able to outperform its unstructured counterpart with considerably fewer parameters.
VII Conclusion
We studied the low separation rank model (LSR-DL) to learn structured dictionaries for tensor data. This model bridges the gap between unstructured and separable dictionary learning (DL) models. For the intractable rank-constrained and the tractable factorization-based LSR-DL formulations, we show that given \Omega\big{(}r(\sum_{n}m_{n}p_{n})p^{2}\rho^{-2}\big{)} data samples, the true dictionary can be locally recovered up to distance . This is a reduction compared to the sample complexity of standard DL in Gribonval et al. [27]. However, a minimax lower bound scaling of in Shakeri et al. [15] for KS-DL () has an gap with our sample complexity upper bound. This gap suggests that the sample complexity bounds may be improved. Possible future directions in this regard include finding minimax bounds for the LSR-DL model and tightening the gap between the sample complexity lower bound (minimax bound) and the upper bounds for this model.
We also show in the regularization-based formulation that samples are sufficient for local identifiability of the true Kronecker-structured (KS) dictionary up to distance . Improving this result and providing sample complexity results for when the true dictionary is LSR (and not just KS) is also another interesting future work.
Finally, we presented two LSR-DL algorithms and showed that they have better generalization performance for image denoising in comparison to unstructured DL algorithm -SVD [5] and existing KS-DL algorithms SeDiL [11] and BCD [29]. We also present OSubDil that to the best our knowledge is the first online algorithm that results in LSR or KS dictionaries. We show that OSubDil results in a faster reduction in the reconstruction error in terms of number of observed samples compared to the state-of-the-art online DL algorithm [6] when the noise level in data is high.
Appendix A The Rearrangement Procedure
To illustrate the procedure that rearranges a KS matrix into a rank- tensor, let us first consider . The elements of can be rearranged to form , where for [10]. Figure 4 depicts this rearrangement for . Similarly, for , we can write , where each frontal slice121212A slice of a -dimensional tensor is a -dimensional section defined by fixing all but two of its indices. For example, a frontal slice is defined by fixing the third index. of the tensor is a scaled copy of . The rearrangement of into is performed via a permutation matrix such that . Given index of and the corresponding mapped index of , our strategy for finding the permutation matrix is to define as a function of . To this end, we first find the corresponding row and column indices of matrix from the th element of . Then, we find the index of the element of interest on the th order rearranged tensor , and finally, we find its location on . Note that the permutation matrix needs to be computed only once in an offline manner, as it is only a function of the dimensions of the factor matrices and not the values of elements of .
We now describe the rearrangement procedure in detail, starting with the more accessible case of KS matrices that are Kronecker product of factor matrices and then extending it to the general case. Throughout this section, we define an -th order “tile” to be a scaled copy of for . A zeroth-order tile is just an element of a matrix. Moreover, we generalize the concept of slices of a rd-order tensor to “hyper-slices”: an -th order hyper-slice is a scaled copy of .
A-A Kronecker Product of Matrices
In the case of 3rd-order tensors, we take the following steps:
- i)
Find index in that corresponds to the -th element of . 2. ii)
Find the corresponding index on the third order tensor . 3. iii)
Find the corresponding index on . 4. iv)
Set .
Let , with and for . For the first operation, we have
[TABLE]
We can see from Figure 2 that the rearrangement procedure works in the following way. For each element indexed by on matrix , find the 2nd-order tile to which it belongs. Let us index this 2nd-order tile by . Then, find the st-order tile (within the nd-order tile indexed ) on which it lies and index this tile by . Finally, index the location of the element (zeroth-order tile) within this first-order tile by . After rearrangement, the location of this element on the rank- tensor is .
In order to find that corresponds to , we first find , then , and then . To find , we need to find the index of the nd-order tile on which the element indexed by lies:
[TABLE]
where and are the number of the nd-order tiles on the left and above the tile to which the element belongs, respectively. Now, we find the position of the element in this nd-order tile:
[TABLE]
For the column index, , we have
[TABLE]
The location of the element on the st-order tile is
[TABLE]
Therefore, can be expressed as
[TABLE]
Finally, in the last step we find the corresponding index on using the following rule.
[TABLE]
This process is illustrated in Figure 2.
A-B The General Case
We now extend our results to -th order tensors. Vectorization and its adjoint operation are easy to compute for tensors of any order. We focus on rearranging elements of to form the -way rank- tensor , where for , , and .
We first formally state the rearrangement and then we explain it. Similar to the case of explained earlier, for each element of the KS matrix indexed by , we first find the th-order tile to which it belongs, then the th-order tile, and so on. Let denote the indices of these tiles, respectively. Then, after rearrangement, the element indexed on KS matrix becomes the element indexed on the rearrangement tensor .
Now, let us find the indices of the tiles of KS matrix to which the element belongs. In the following, we denote by the index of this element within its th-order tile. Note that since is an th-order tile itself, we can use instead of to refer to the index of the element on for consistency of notation. For the -th element of we have
[TABLE]
where is the index of the -th order tile and is the location of the given element within this tile. Similarly, we have
[TABLE]
for . Finally, we have
[TABLE]
It is now easy to see that the -th element of is the -th element of .
Intuitively, notice that -th order KS matrix is a tiling of KS tiles of order . In rearranging into , the elements of each of these -th order tiles construct a -th order “hyper-slice”. On matrix , these tiles consist of tiles, each of which is a th-order KS matrix, whose elements are rearranged to a -th hyper-slice of , and so on. Hence, the idea is to use the correspondence between the th-order tiles and th-order hyper-slices: finding the index of the -th order tile of on which lies is equivalent to finding the index of the th-order hyper-slice of to which it is translated. Note that each entry of a tensor in indexed by an -tuple and the index of an entry of a tensor on its th hyper-slice is in fact its th element in the index tuple of this entry. Therefore, we first find the -th order KS tile of on which the element lies (equivalent to finding the th-order hyper-slice to which is translated), and then find the location of this element on this tile. Next, the -th order KS tile in which lies is found as well as the location of the element within this tile, and so on.
Appendix B Proofs of Lemmas
Proof of Lemma 2.
Proposition 4.1 in De Silva and Lim [41] shows that the space of tensors of order and rank is not closed. The fact that the rearrangement process preserves topological properties of sets means that the same result holds for the set with and rank .
The proof for closedness of and follows from Propositions 4.2 and 4.3 in De Silva and Lim [41], which can be adopted here due to the relation between the sets of low-rank tensors and LSR matrices. ∎
Proof of Lemma 3.
The rearrangement process allows us to borrow the results in Proposition 4.8 in De Silva and Lim [41] for tensors and apply them to LSR matrices. ∎
Proof of Lemma 6.
Define and . Since the rearrangement operator is an isometry w.r.t. the Euclidean distance, the image of an -net of w.r.t. the Frobenius norm under this rearrangement operator is an -net of ( and ) w.r.t the Frobenius norm. Thus, We also know that [46]. This means that
[TABLE]
On the other hand, for the oblique manifold we have and therefore, . Hence, Also, since for any , it follows that an -covering of any given set w.r.t. the Frobenius norm is also an -covering of that set w.r.t. the max-column-norm. Thus Moreover, it follows from the fact that
[TABLE]
Thus, from (35) and (36) we see that , which concludes the proof. ∎
Proof of Lemma 7.
Each element can be written as a summation of at most KS matrices such that . This implies that is a subset of the Minkowski sum (vector sum) of copies of , the set of KS matrices within the Euclidean ball of radius . It is easy to show that the Minkowski sum of the -coverings of sets is an -covering of the Minkowski sum of those sets in any norm. Therefore, we have
[TABLE]
Moreover, we have . We also know from equation (16) that . Putting all these facts together, we get
[TABLE]
∎
Proof of Lemma 8.
According to Lemma 2 in Shakeri et al. [16], for any and we have
[TABLE]
where or depending on . Let . Using equality (B), we have
[TABLE]
where the inequality follows from . ∎
Appendix C Discussion on Convergence of the Algorithms
The batch algorithms proposed in Section V are essentially variants of alternating minimization (AM). Establishing the convergence of AM-type algorithms in general is challenging and only known for limited cases. Here, we first present a well-known convergence result for AM-type algorithms in Lemma 9 and discuss why our algorithms STARK and TeFDiL do not satisfy the requirements of this lemma. Then, we show a possible approach for proving convergence of STARK. We do not discuss convergence analysis of OSubDil here since it does not fall in the batch AM framework that we discuss here. We leave formal convergence results of our algorithms as open problems for future work.
First, let us state the following standard convergence result for alternating minimization-type algorithms.
Lemma 9** (Proposition 2.7.1, [47]).**
Consider the problem
[TABLE]
where are closed convex subsets of the Euclidean space. Assume that is a continuous differentiable over the set . Suppose for each and all , the minimum
[TABLE]
is uniquely attained. Then every limit point of the sequence generated by block coordinate descent method is a stationary point of .
The result of Lemma 9 cannot be used for TeFDiL since its dictionary update stage does not have a unique minimizer (nonconvex minimization problem with multiple global minima)). Moreover, as discussed in Section V-B, TeFDiL only returns an inexact solution in the dictionary update stage.
Similarly, this result cannot be used to show convergence of STARK to a stationary point of Problem (8) since, as discussed in Section V-A, STARK returns an inexact solution in the dictionary update stage. However, we show next that dropping the unit column-norm constraint allows us to provide certain convergence guarantees. The unit column-norm constraint is essential in standard DL algorithms since in its absence, the norm regularization term encourages undesirable solutions where is very small while is very large. However, in the regularization-based LSR-DL problem, the additional regularization term ensures this does not happen. Therefore, dropping the unit column-norm constraint is sensible in this problem.
Let us discuss what guarantees we are able to obtain after relaxing the constraint set . Consider the problem
[TABLE]
We show in Proposition 1 that under the following assumptions, STARK converges to a stationary point of Problem (41) (when the normalization step is not enforced). Then we discuss how this problem is related to Problem (8).
Assumption 4**.**
Consider the sequence \big{(}\mathbf{D}(t),\mathbf{X}(t)\big{)} generated by STARK. We assume that for all :
- I)
Classical optimality conditions for the lasso problem (see Tibshirani **[48]**) are satisfied. 2. II)
* is full row-rank at all .*
Proposition 1**.**
Under Assumption 4, STARK converges to a stationary point of problem (41).
Proof.
We invoke Lemma 9 to show the convergence of STARK. To use this lemma, the minimization problem w.r.t. each block needs to correspond to a closed convex constraint set and also needs to have a unique minimizer.
In the sparse coding stage, given Assumption 4-I, the minimizer of the lasso problem is unique. In the dictionary update stage of STARK, the objective of problem (41) is strongly convex w.r.t. under Assumption 4-II and thus has a unique minimizer. Moreover, the constraint set is closed and convex. To utilize Lemma 9, it remains to show that this minimum is actually attained by ADMM. To this end, we restate Problem (V-A) as
[TABLE]
where ( is a linear function of ) and . It is clear that is convertible. Therefore, according to [49][Chapter 3, Proposition 4.2], the ADMM algorithm converges to the unique minimizer of Problem (V-A).
∎
So far we discussed convergence of STARK to Problem (41) while our identifiability results are for problem (8). There is, however, a strong connection between minimization Problems (8) and (41): for each local minimum of problem (8), there exists an such that is a local minimum of (41). Define . Consider any that is a local minimum of (8) and let . We have . Since is a local minimizer of , for any in the local neighborhood of . Also by definition, for any . Thus, for any in the local neighborhood of , meaning that is a local minimizer of (41). Since we showed in Section IV that a local minimum of (8) is close to the true dictionary , we can say there is a local minimum of (41) close to . So our recovery result for (8) can apply to our proposed algorithm for solving (41) as well.
The reference list from the paper itself. Each links out to its DOI / PubMed record.
- 1[1] M. Ghassemi, Z. Shakeri, A. D. Sarwate, and W. U. Bajwa, “STARK: Structured dictionary learning through rank-one tensor recovery,” in Proc. IEEE 7th Int. Workshop Computational Advances in Multi-Sensor Adaptive Processing (CAMSAP) , 2017, pp. 1–5.
- 2[2] M. Ghassemi, Z. Shakeri, W. U. Bajwa, and A. D. Sarwate, “Sample complexity bounds for low-separation-rank dictionary learning,” in Proc. 2019 IEEE Int. Symp. Inf. Theory , July 2019.
- 3[3] K. Kreutz-Delgado, J. F. Murray, B. D. Rao, K. Engan, T.-W. Lee, and T. J. Sejnowski, “Dictionary learning algorithms for sparse representation,” Neural Computation , vol. 15, no. 2, pp. 349–396, 2003.
- 4[4] I. Tosic and P. Frossard, “Dictionary learning,” IEEE Signal Processing Magazine , vol. 28, no. 2, pp. 27–38, March 2011.
- 5[5] M. Aharon, M. Elad, and A. Bruckstein, “ K 𝐾 K -SVD: An algorithm for designing overcomplete dictionaries for sparse representation,” IEEE Trans. Signal Process. , vol. 54, no. 11, pp. 4311–4322, November 2006.
- 6[6] J. Mairal, F. Bach, J. Ponce, and G. Sapiro, “Online learning for matrix factorization and sparse coding,” J. Machine Learning Research , vol. 11, pp. 19–60, 2010.
- 7[7] L. R. Tucker, “Implications of factor analysis of three-way matrices for measurement of change,” Prob. Meas. Change , pp. 122–137, 1963.
- 8[8] R. A. Harshman, “Foundations of the PARAFAC procedure: Models and conditions for an explanatory multi-modal factor analysis,” UCLA Working Papers in Phonetics , vol. 16, pp. 1–84, 1970.
