Deep Sparse Representation-based Classification

TL;DR

This paper introduces a deep learning model combining autoencoders and sparse coding for improved classification, demonstrating superior results over traditional SRC methods across multiple datasets.

Contribution

It proposes a novel deep transductive framework that integrates deep feature learning with sparse representation for classification.

Findings

Outperforms state-of-the-art SRC methods on three datasets

Learns robust deep features for sparse coding

Provides open-source implementation for reproducibility

Abstract

We present a transductive deep learning-based formulation for the sparse representation-based classification (SRC) method. The proposed network consists of a convolutional autoencoder along with a fully-connected layer. The role of the autoencoder network is to learn robust deep features for classification. On the other hand, the fully-connected layer, which is placed in between the encoder and the decoder networks, is responsible for finding the sparse representation. The estimated sparse codes are then used for classification. Various experiments on three different datasets show that the proposed network leads to sparse representations that give better classification results than state-of-the-art SRC methods. The source code is available at: github.com/mahdiabavisani/DSRC.

Figures

Tables

Table 1. TABLE I: Details of our networks. Note that the number of parameters in the sparse coding layer rely on the size of dataset including the n 𝑛 n training and m 𝑚 m test samples.

	Layer	Input	Output	Kernel	(stride, pad)
Encoder	Conv 1	$𝐗$	Conv 1	$1\times 5\times 5\times 10$	(2,1)
	Conv 2	Conv 1	Conv 2	$1\times 3\times 3\times 20$	(2,1)
	Conv 3	Conv 2	Conv 3	$1\times 3\times 3\times 30$	(1,0)
	Conv 4	Conv 3	$𝐙$	$1\times 3\times 3\times 30$	(1,0)
Sparse coding layer	${𝚯}_{sc}$	$𝐙$	$\widehat{𝐙}$	$m\times n$ Parameters	-
Decoder	deconv 1	$\widehat{𝐙}$	deconv 1	$1\times 3\times 3\times 30$	(1,0)
	deconv 2	deconv 1	deconv 2	$1\times 3\times 3\times 20$	(2,1)
	deconv 3	deconv 2	$\widehat{𝐗}$	$1\times 5\times 5\times 10$	(2,1)

Table 2. TABLE II: Sparse representation-based classification accuracy of different methods.

Dataset	SRC	KSRC	AE-SRC	VGG19-SRC	InceptionV3-SRC	Resnet50-SRC	Denesnet169-SRC	DSRC (ours)
USPS	87.78	91.34	88.65	91.27	93.51	95.75	95.26	96.25
SVHN	15.71	27.42	18.69	52.86	41.14	47.88	37.65	67.75
UMDAA-01	79.00	81.37	86.70	82.68	86.15	91.84	86.35	93.39

Table 3. TABLE III: The classification accuracy corresponding to the ablation study. N/C refers to the cases where the learning process did not converge.

	DSRC	DSC-SRC	DSRC0.5	DSRC1.5	DSRC2
USPS	96.25	78.25	N/C	95.75	96.25

Equations

$${\mathbf{X}}_{train}=[{\mathbf{X}}_{train}^1,{\mathbf{X}}_{train}^2,\cdots ,{\mathbf{X}}_{train}^K]\in {\mathbb{R}}^{d_0\times n}$$

$$\underset{\mathbf{\alpha }}{\min }\Vert \mathbf{\alpha }{\Vert }_0\text{s.t.}{\mathbf{x}}_{test}={\mathbf{X}}_{train}\mathbf{\alpha },$$

$$\underset{\mathbf{\alpha }}{\min }\Vert {\mathbf{x}}_{test}-{\mathbf{X}}_{train}\mathbf{\alpha }{\Vert }_2^2+{\lambda }_0\Vert \mathbf{\alpha }{\Vert }_1,$$

$$\text{class}({\mathbf{x}}_{test})=\text{arg}\underset{k}{\min }\Vert {\mathbf{x}}_{test}-{\mathbf{X}}_{train}{\delta }_k(\mathbf{\alpha }){\Vert }_2^2,$$

$$\underset{\mathbf{A}}{\min }\Vert {\mathbf{Z}}_{test}-{\mathbf{Z}}_{train}\mathbf{A}{\Vert }_F^2+{\lambda }_0\Vert \mathbf{A}{\Vert }_1,$$

$${\hat{\mathbf{Z}}}_{train}={\mathbf{Z}}_{train}{\mathbf{I}}_n,{\hat{\mathbf{Z}}}_{test}={\mathbf{Z}}_{train}\mathbf{A},$$

$${\mathbf{\Theta }}_{sc}=\left[\begin{array}{c}{\mathbf{I}}_n\text{hfill}\mathbf{A}\text{hfill}\\ 0_{n\times m}{0}_m\text{hfill}\end{array}\right].$$

$$\underset{\mathbf{\Theta }}{\min }\Vert \mathbf{Z}-\mathbf{Z}{\mathbf{\Theta }}_{sc}{\Vert }_F^2+{\lambda }_0\Vert {\mathbf{\Theta }}_{sc}{\Vert }_1+{\lambda }_1\Vert \mathbf{X}-\hat{\mathbf{X}}{\Vert }_F^2,$$

$$\text{class}({\mathbf{x}}_{test}^i)=\text{arg}\underset{k}{\min }\Vert {\mathbf{z}}_{test}^i-{\mathbf{Z}}_{train}{\delta }_k({\mathbf{\alpha }}^i){\Vert }_2^2.$$

Full text

I Introduction

Sparse coding has become widely recognized as a powerful tool in signal processing and machine learning with various applications in computer vision and pattern recognition [1, 2, 3]. Sparse representation-based classification (SRC) as an application of sparse coding was first proposed in [1], and was shown to provide robust performance on various face recognition datasets. Since then, SRC has been used in numerous applications [4, 5, 6, 7, 8, 9]. In SRC, an unlabeled sample is represented as a sparse linear combination of the labeled training samples. This representation is obtained by solving a sparsity-promoting optimization problem. Once the representation is found, the label is assigned to the test sample based on the minimum reconstruction error rule [1].

The SRC method is based on finding a linear representation of the data. However, linear representations are almost always inadequate for representing non-linear structures of the data which arise in many practical applications. To deal with this issue, some works have exploited the kernel trick to develop non-linear extensions of the SRC-based methods [10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20]. Kernel SRC methods require the use of a pre-determined kernel function such as polynomial or Gaussian. Selection of the kernel function and its parameters is an important issue in training when kernel SRC methods are used for classification.

In this paper, we propose a deep neural network-based framework that finds an explicit nonlinear mapping of data, while simultaneously obtaining sparse codes that can be used for classification. Learning nonlinear mappings with neural networks has been shown to produce remarkable improvements in subspace clustering tasks [21, 22]. We introduce a transductive model, which accepts a set of training and test samples, learns a mapping that is suitable for sparse representation, and recovers the corresponding sparse codes. Our model consists of an encoder that is responsible for learning the mapping, a sparse coding layer which mimics the task of constructing the mapped test samples by a combination of the mapped training samples, and a decoder that is used for training the networks.

I-A Sparse representation-based classification

In SRC, given a set of labeled training samples, the goal is to classify an unseen set of test samples. Suppose that we collect all the vectorized training samples with the label $i$ in the matrix $\mathbf{X}^{i}_{train}\in\mathbb{R}^{d_{0}\times{n_{i}}}$, where $d_{0}$ is the dimension of each sample and $n_{i}$ is the number of samples in class $i$, then the training matrix can be constructed as

[TABLE]

where $n_{1}+n_{2}+\cdots+n_{K}=n$ and we have a total of $K$ classes.

In SRC, it is assumed that an observed sample $\mathbf{x}_{test}\in\mathbb{R}^{d_{0}}$ can be well approximated by a linear combination of the samples in $\mathbf{X}^{i}_{train}$ if $\mathbf{x}_{test}$ is from class $i$. Thus, it is possible to predict the class of a given unlabeled data by finding a set of samples in the training set that can better approximate $\mathbf{x}_{test}$. Mathematically, these samples can be found by solving the following optimization problem

[TABLE]

where $\|\boldsymbol{\alpha}\|_{0}$ counts the number of non-zero elements in $\boldsymbol{\alpha}$. The minimization problem (2) finds a sparse solution for the linear system. However, since the optimization problem (2) is an NP-hard problem, in practice, a sparsity constraint is enforced by the $\ell_{1}$-norm of $\boldsymbol{\alpha}$ which is a convex relaxation of the above problem  [23, 24]. Thus, in practice the following minimization problem is solved to obtain the sparse codes

[TABLE]

where $\lambda_{0}$ is a positive regularization parameter. Once $\boldsymbol{\alpha}$ is found, one can estimate the class label of $\mathbf{x}_{test}$ as follows

[TABLE]

where $\delta_{k}(\cdot)$ is the characteristic function that selects the coefficients associated with the class $i$.

II Deep sparse representation-based classification network

We develop a transductive classification model based on sparse representations. In a transductive model, as opposed to inductive models, both training and test sets are observed, and the learning process pursues reasoning from the specific training samples to a specific set of test cases [25]. We build our method based on convolutional autoencoders. In particular, our network contains an encoder, a sparse coding layer, and a decoder. The encoder receives both the training and test sets as raw data inputs and extracts abstract features from them. The sparse coding layer recovers the test cases by a sparse linear combination of the training samples, and concatenates them along with the training features which are then fed to the decoder. The decoder maps both the training embeddings and the recovered test embeddings back to the original representation of the data. Figure 1 gives an overview of the proposed deep SRC (DSRC) framework.

**Sparse representation: ** Let $\mathbf{X}_{train}\in\mathbb{R}^{d_{0}\times n}$ and, $\mathbf{X}_{test}\in\mathbb{R}^{d_{0}\times m}$ be the given vectorized training and testing data, respectively. We feed $\mathbf{X}=[\mathbf{X}_{train},\mathbf{X}_{test}]$ to the encoder, where it develops the corresponding embedding features $\mathbf{Z}=[\mathbf{Z}_{train},\mathbf{Z}_{test}]\in\mathbb{R}^{d_{z}\times(m+n)}$. The minimization problem (3) for a single test observation can be re-written for a matrix of testing embedding features as

[TABLE]

where $\mathbf{A}\in R^{n\times m}$ is the coefficient matrix that contains the sparse codes in its columns, and $\lambda_{0}$ is a positive regularization parameter. Note that the first penalty term in equation (5) is equivalent to the penalty term used for a fully-connected neural network layer with the input of $\mathbf{Z}_{train}$, the output of $\mathbf{Z}_{test}$ and trainable parameters of $\mathbf{A}$. As a result, considering the sparsity constraint, one can model the optimization problem (5) in a neural network framework with a fully-connected layer with sparse parameters which have no non-linearity activation or bias nodes. We use such a model inside our sparse coding layer to find the sparse codes for the observed test set.

The sparse coding layer is located between the encoder and decoder networks. Its task for $\mathbf{Z}_{train}$ is to pass them to the decoder, and for the test features $\mathbf{Z}_{test}$ it will pass their reconstructions that are found from (5), as $\mathbf{Z}_{train}\mathbf{A}$, to the decoder. Thus, assuming that $\mathbf{\hat{Z}}_{train}$ and $\mathbf{\hat{Z}}_{test}$ are the outputs of the sparse coding layer for training and testing features, we have

[TABLE]

where $\mathbf{I}_{n}\in\mathbb{R}^{n\times n}$ is the identity matrix. Therefore, if the decoder’s input is $\mathbf{\hat{Z}}=[\mathbf{\hat{Z}}_{train},\mathbf{\hat{Z}}_{test}]$, from (6) we can calculate $\mathbf{\hat{Z}}$ as $\mathbf{\hat{Z}}=\mathbf{Z}\boldsymbol{\Theta}_{sc}$, where

[TABLE]

In equation (7), $\mathbf{0}_{n\times m}\in\mathbb{R}^{n\times m}$ and $\mathbf{0}_{m}\in\mathbb{R}^{m\times m}$ are zero matrices. One can write an end-to-end training objective that includes sparse coding and training of the encoder-decoder as follows

[TABLE]

where $\boldsymbol{\Theta}$ is the union of all the trainable parameters including encoder and decoder’s parameters and $\mathbf{A}$. Here, $\mathbf{\hat{X}}=[\mathbf{\hat{X}}_{train},\mathbf{\hat{X}}_{test}]$ is the output of the decoder (i.e. reconstructions), and $\lambda_{0}$ and $\lambda_{1}$ are positive regularization parameters. Note that the optimization problem (8) simultaneously finds sparse codes $\mathbf{A}$ and a set of desirable embedding features $\mathbf{Z}$ that are especially suitable for providing efficient sparse codes.

Classification: Once the sparse coefficient matrix $\mathbf{A}$ is found, it can be used for associating the class labels to the test samples. For each test sample $\mathbf{x}^{i}_{test}$ in $\mathbf{X}_{test}$, its embedding features $\mathbf{z}^{i}_{test}$, and the corresponding sparse code column $\boldsymbol{\alpha}^{i}$ in $\mathbf{A}$ are used to estimate the class labels as follows

[TABLE]

The proposed DSRC method is summarized in Algorithm 1.

III Experimental results

In this section, we evaluate our method against state-of-the-art SRC methods. The USPS handwritten digits dataset [26], the street view house numbers (SVHN) dataset [27], and the UMDAA-01 face recognition dataset [28] are used in our experiments. Figure 2 (a), (b), and (c) show sample images from these datasets. Since the number of parameters in the sparse coding layer scales with the multiplication of training and testing sizes, we randomly select a smaller subset of the used datasets and perform all the experiments on the selected subset. In all the experiments, the input images are resized to $32\times 32$.

We compare our method with the standard SRC method [1], Kernel SRC (KSRC) [14], SRC on features extracted from an autoencoder with similar architecture to our network (AE-SRC), and SRC on features extracted from the state-of-the-art pre-trained networks. In our experiment with the pre-trained networks, the networks are pre-trained on the Imagenet dataset [29]. For this purpose, we use the following four popular network architectures: VGG-19[30], Inception-V3 [31], Resnet-50 [32] and Densenet-169 [33]. We feed these networks with our datasets, extract the features corresponding to the last layer before classification, and pass them to the classical SRC algorithm. Note these networks accept $224\times 224$ inputs. Thus, as a preprocessing step, we resample the input images to $224\times 224$ images before feeding them to the pre-trained networks.

We compare different methods in terms of their five-fold averaged classification accuracy. In all the experiments, unless otherwise stated, we randomly split the datasets into sets of training and testing samples, where $20\%$ of the samples are used for testing, and $80\%$ of the samples are used as the training set.

Network structure: The encoder network of our model consists of stacked four convolutional layers, and the decoder has three fractionally-strided convolution layers (also known as deconvolution layers). Each plugged in convolution or fractionally-strided convolution is coupled with a ReLu nonlinearity as well, but does not have a batch-norm layer. Table I gives the details of the network, including the kernel sizes and the number of filters.

Training details: We implemented our method with Tensorflow-1.4 [34]. We use the adaptive momentum-based gradient descent method (ADAM) [35] to minimize the loss function, and apply a learning rate of $10^{-3}$. Before we start training on our objective function, in each experiment, we pre-train our encoder and decoder on the dataset without the sparse coding layer. In particular, we pre-train the encoder-decoder for $20k$ epochs with the objective of $\min_{\hat{\boldsymbol{\Theta}}}\|\mathbf{X}-\hat{\mathbf{X}}\|^{2}_{F},$ where $\hat{\boldsymbol{\Theta}}$ indicates the union of parameters in the encoder and decoder networks. We use a batch size of $100$ for this stage. However, in the actual stage of training, we feed all the samples including the training and testing samples as a single large batch. We set the regularization parameters as $\lambda_{0}=1$ and $\lambda_{1}=8$ in all the experiments.

III-A USPS digits

The first set of experiments is conducted on the USPS handwritten digits dataset [26]. This dataset contains 7291 training and 2007 test grayscale images of ten digits (0-9). Figure 2 (a), shows example images from this dataset. We perform the experiments on a subset with a total size of $2000$ samples. In particular, we randomly select $160$ and $40$ samples per digit from the training and testing sets, respectively. The first row of Table II shows the performance of various SRC methods. As can be observed from this table, the proposed method performs significantly better than the other methods including the classical and deep learning-based methods.

Figure 3 shows the coefficient matrix $\mathbf{A}$, extracted from $\boldsymbol{\Theta_{sc}}$, the matrix of the network trained for this experiment. For better visualization, we show the absolute value of the transposed $\mathbf{A}$ (i.e. $|\mathbf{A}^{T}|$). Thus, each row of the matrix in Figure 3 corresponds to the sparse codes for one of the test samples. Similarly, columns in this figure are coefficients related to the training samples. This matrix is sparse and shows a block diagram pattern, where most of the non-zero coefficients for each test sample are those that correspond to the training samples with the same class as the observed test sample.

Analysis of the network: To understand the effects of some of our model choices, we compare the performance of our DSRC method with variations of it by changing the regularization norm on $\boldsymbol{\Theta_{sc}}$ in the loss function (8). We replace the term $\|\boldsymbol{\Theta}_{sc}\|_{1}$ in (8) by $\|\boldsymbol{\Theta}_{sc}\|_{p}$, where $p=0.5,1.5$ and $2$, and report their performances by DSRC0.5, DSRC1.5 and DSRC2, respectively.

In addition, if we do not follow the specific structure described in equation (7), and instead have a fully connected layer with $(m+n)^{2}$ parameters which receives $\mathbf{Z}$ and reconstructs $\mathbf{\hat{Z}}$, the architecture of the network will be similar to the deep subspace clustering networks (DSC) proposed in [21] for the task of subspace clustering. As an ablation study, we use this method to extract sparse codes and then apply the same classification rule as in (9) to estimate class labels for the test set. We call this method DSC-SRC.

Table III reveals that while the regularization norm on the coefficient matrix is selected between $\ell_{1}$ and $\ell_{2}$, it does not have much effect on the performance of the classification task. However, in our experiments, we observed that for norms smaller than 1, the problem is not stable and often does not converges. In addition, DSC-SRC cannot provide a desirable performance. Note that the fully-connected layer in this method (counterpart to our sparse coding layer) does not limit the testing features to be reconstructed with only the training features. As a result, it is possible that testing features shape an isolated group that does not have a strong connection to the training features. This makes it more difficult to estimate a label for the test samples.

III-B Street view house numbers

The SVHN dataset [27] contains 630,420 color images of real-world house numbers collected from Google Street View images. This dataset has three splits as the training set with 73,257 images, the testing set with 26,032 images and an extra set containing 531,131 additional samples. In this experiment, similar to our experiments on MNIST, we randomly select 160 images per digit from the training split and 40 per digit from the test split. This dataset is much more challenging than MNIST. This is in part due to the large variations of data. Furthermore, many samples in this dataset contain multiple digits in an image. The task is to classify the center digit.

The second row in Table II compares the performance of different SRC methods. This table demonstrates the advantage of our method. While the classification task is much more challenging on SVHN than MNIST, the gap between the performance of our method and the second best performance is even more. The next best performing method is VGG19-SRC which performs $14.86\%$ behind the accuracy of our method.

III-C UMD mobile faces

The UMD mobile face dataset (UMDAA-01) [28] contains 750 front-facing camera videos of 50 users captured while using a smartphone. This dataset has been collected over three different sessions. This dataset was originally collected for the active authentication task, but since its frames include challenging facial image instances with various illumination and pose conditions it has also been used for other tasks [36, 37]. In this experiment, we randomly select 50 facial images per subjects from the data in Session 1. Figure 2 shows some sample images from this dataset.

The performance of various SRC methods on the UMDAA-01 dataset are tabulated in the third row of Table II. As can be seen, our proposed DSRC method similar to the experiments with SVHN provides remarkable improvements as compared to the other SRC methods. This clearly shows that more challenging datasets are better represented by our method. This is because our method not only efficiently finds the sparse codes, but also it seeks for a representation of data (the output of the encoder) that is especially suitable for sparse representation.

Comparison to state-of-the-art classification networks: While deep neural networks perform very well when they are trained on large datasets, in the case of limited number of labeled training samples, they often tend to overfit to the training samples. The objective of this experiment is to analyze the performance of our approach in such circumstances. We compare our method to the following classification networks: VGG-19[30], Inception-V3 [31], Resnet-50 [32] and Densenet-169 [33]. We first pre-train the networks on the Imagenet dataset [29], and then fine-tune them on the available training samples in UMDAA-01.

Figure 4 shows the performance of the classification networks on four different versions of UMDAA-01 dataset with varying number of training samples. The four versions are created by randomly splitting the dataset into sets of training and testing samples that respectively contain $20\%$, $40\%$, $60\%$ and $80\%$ of the total number of samples as training samples and use the rest of samples as the testing set. As the figure suggests, accuracy improves by increasing the number of training samples in all the cases. However, the results show better performances for DSRC even when less training data is available.

IV Conclusion

We presented an autoencoder-based sparse coding network for SRC. In particular, we introduced a sparse coding layer that is located between the conventional encoder and decoder networks. This layer recovers sparse codes of embedding features that are received from the encoder. The spare codes are later used to estimate the class labels of testing samples. We discussed a framework that allows an end-to-end training. Various experiments on three different image classification datasets showed that the proposed network leads to sparse representations that give better classification results than state-of-the-art SRC methods.

Appendix: Convergence

To empirically show the convergence of our proposed method, in Figure 5, we plot the values of the objective function of DSRC in the experiment with the UMDAA-01 dataset and its classification accuracy in different iterations. The reported loss values in Figure 5 are scaled to have a maximum value of one. As can be seen from the figure, our algorithm converges in a few iterations.