Generative approach to unsupervised deep local learning

TL;DR

This paper introduces a novel generative deep learning framework that pretrains convolutional autoencoders, constructs affinity graphs, and employs a self-expressive layer with a locality-preserving loss to enhance unsupervised feature learning.

Contribution

It proposes a new generative approach combining autoencoders, affinity graph construction, and a self-expressive layer with a locality-preserving loss for improved unsupervised feature learning.

Findings

Outperforms state-of-the-art methods on four datasets

Effectively preserves local data structure during learning

Enhances the quality of learned feature representations

Abstract

Most existing feature learning methods optimize inflexible handcrafted features and the affinity matrix is constructed by shallow linear embedding methods. Different from these conventional methods, we pretrain a generative neural network by stacking convolutional autoencoders to learn the latent data representation and then construct an affinity graph with them as a prior. Based on the pretrained model and the constructed graph, we add a self-expressive layer to complete the generative model and then fine-tune it with a new loss function, including the reconstruction loss and a deliberately defined locality-preserving loss. The locality-preserving loss designed by the constructed affinity graph serves as prior to preserve the local structure during the fine-tuning stage, which in turn improves the quality of feature representation effectively. Furthermore, the self-expressive layer between the encoder and decoder is based on the assumption that each latent feature is a linear combination of other latent features, so the weighted combination coefficients of the self-expressive layer are used to construct a new refined affinity graph for representing the data structure. We conduct experiments on four datasets to demonstrate the superiority of the representation ability of our proposed model over the state-of-the-art methods.

Tables

Table 1. Table 1: Comparison of the recently proposed unsupervised learning methods with our UDLL approach. CNN denotes convolutional neural network, SL denotes subspace learning, and LP denotes the locality preserving.

	Tian et al.[29]	Ji et al.[12]	Xie et al.[35]	Guo et al.[9]	Chang et al.[5]	Ours
CNN	$\times$	✓	$\times$	$\times$	✓	✓
SL	$\times$	✓	$\times$	$\times$	$\times$	✓
LP	$\times$	$\times$	$\times$	✓	$\times$	✓

Table 2. Table 2: Dataset description.

Dataset	# Image	# Class	Image size
COIL-20	1440	20	$32\times 32$
COIL-100	7200	100	$32\times 32$
ORL	400	40	$32\times 32$
Yale	2432	38	$42\times 42$

Table 3. Table 3: Clustering accuracy of different methods (ACC %).

	LRR	LRSC	SSC	KSSC	EDSC	SSC-OMP	DEC	DSC	UDLL
COIL-20	68.99	68.75	85.14	75.35	85.14	54.10	79.00	94.86	97.57
COIL-100	40.18	49.33	55.00	52.82	61.87	33.61	60.66	69.04	70.86
ORL	61.75	67.50	67.70	65.75	72.75	64.00	60.33	86.00	87.75
Extend Yale B	65.13	70.11	72.49	72.25	88.36	75.29	48.66	97.33	97.70

Table 4. Table 4: Clustering accuracy of different features (ACC %).

	PT$+$SSC	PT$+$EDSC	UDLL
COIL-20	77.92	84.21	97.57
COIL-100	56.07	61.12	70.86
ORL	73.25	73.75	87.75
Extend Yale B	74.67	87.36	97.70

Equations

$$\mathbf{L}({\mathbf{\Theta }}_e,{\mathbf{\Theta }}_d)=\frac{1}{2}\Vert \mathbf{X}-\hat{\mathbf{X}}{\Vert }_{\mathrm{F}}^2$$

$$\begin{array}{rl}\underset{\mathbf{A}}{\min }& \sum _{i,j=1}^n\Vert {\mathbf{z}}_i-{\mathbf{z}}_j{\Vert }_2^2{a}_{ij}+\lambda \Vert \mathbf{A}{\Vert }_{\mathrm{F}}^2\\ s.t.& \text{leavevmode}\mathbf{A}\ge 0,\text{leavevmode}\mathbf{A}1=1.\end{array}$$

$$\begin{array}{rl}& \underset{{\mathbf{a}}_j}{\min }\sum _{i=1}^n\Vert {\mathbf{z}}_i-{\mathbf{z}}_j{\Vert }_2^2{a}_{ij}+\lambda \sum _{i=1}^n{a}_{ij}^2\\ & \text{leavevmode}\text{leavevmode}s.t.\text{leavevmode}{\mathbf{a}}_j\ge 0,\text{leavevmode}{1}^{\top }{\mathbf{a}}_j=1.\end{array}$$

$$\begin{split}\min_{{\bm{a}}_{j}}&\frac{1}{2}\Bigl{\|}{\bm{a}}_{j}+\frac{1}{2\lambda}{\bm{m}}\Bigr{\|}_{2}^{2}\\ \leavevmode\nobreak\ \leavevmode\nobreak\ {\rm s.t.}&\leavevmode\nobreak\ {\bm{a}}_{j}\geq 0\,,\leavevmode\nobreak\ \bm{1}^{{\top}}{\bm{a}}_{j}=1\,.\end{split}$$

$$\begin{split}\mathcal{L}\left({\bm{a}}_{j}\,,\eta\,,\bm{\rho}\right)=&\Bigl{\|}{\bm{a}}_{j}+\frac{1}{2\lambda}{\bm{m}}\Bigr{\|}_{2}^{2}\\ &-\eta(\bm{1}^{{\top}}{\bm{a}}_{j}-1)-\bm{\rho}^{{\top}}{\bm{a}}_{j}\end{split}$$

$${\mathbf{a}}_j^{\star }$$

$$1^{\top }{\mathbf{a}}_j^{\star }$$

$${\mathbf{\rho }}^{\star }$$

$${\rho }_i^{\star }{a}_{ij}^{\star }$$

$$a_{ij}^{\star }+\frac{m_{ij}}{2\lambda }-{\rho }_i^{\star }-{\eta }^{\star }$$

$${\eta }^{\star }\le a_{ij}^{\star }+\frac{m_{ij}}{2\lambda },\forall i\in \{1,2,\dots ,n\}.$$

$$a_{ij}^{\star }(a_{ij}^{\star }+\frac{m_{ij}}{2\lambda }-{\eta }^{\star })=0,\forall i\in \{1,2,\dots ,n\}.$$

$$a_{ij}^{\star }=\{\begin{array}{ll}-\frac{m_{ij}}{2\lambda }+{\eta }^{\star },& \mathrm{if}\text{leavevmode}{\eta }^{\star }>\frac{m_{ij}}{2\lambda };\\ \text{leavevmode}\text{leavevmode}\text{leavevmode}\text{leavevmode}\text{leavevmode}\text{leavevmode}\text{leavevmode}\text{leavevmode}0,& \mathrm{if}\text{leavevmode}{\eta }^{\star }\le \frac{m_{ij}}{2\lambda }.\end{array}$$

$$\sum _{i=1}^k\left(-\frac{m_{ij}}{2\lambda }+{\eta }^{\star }\right)=1\Rightarrow {\eta }^{\star }=\frac{2\lambda +{\sum }_{i=1}^k{m}_{ij}}{2k\lambda }.$$

$$\frac{k}{2}{m}_{kj}-\frac{1}{2}\sum _{i=1}^k{m}_{ij}<\lambda \le \frac{k}{2}{m}_{k+1,j}-\frac{1}{2}\sum _{i=1}^k{m}_{ij}.$$

$$\lambda =\frac{k}{2}{m}_{k+1,j}-\frac{1}{2}\sum _{i=1}^k{m}_{ij}.$$

$${\mathbf{a}}_{ij}^{\star }=\{\begin{array}{ll}\frac{m_{k+1,j}-m_{ij}}{k{m}_{k+1,j}-{\sum }_{p=1}^k{m}_{pj}},& \mathrm{if}\text{leavevmode}i\le k;\\ \text{leavevmode}\text{leavevmode}\text{leavevmode}\text{leavevmode}\text{leavevmode}\text{leavevmode}\text{leavevmode}\text{leavevmode}0,& \mathrm{otherwise}.\end{array}$$

$$\begin{array}{rl}\mathbf{L}({\mathbf{\Theta }}_e,\mathbf{W},{\mathbf{\Theta }}_d)=& \underset{{\mathbf{L}}_{\mathrm{reconstruction}}}{\underbrace{\frac{1}{2}\Vert \mathbf{X}-\hat{\mathbf{X}}{\Vert }_{\mathrm{F}}^2}}+\underset{{\mathbf{L}}_{\mathrm{affinity}}}{\underbrace{\alpha \Vert \mathbf{Z}-\mathbf{Z}\mathbf{W}{\Vert }_{\mathrm{F}}^2+\beta \Vert \mathbf{W}{\Vert }_{\mathrm{F}}^2}}\\ & +\underset{{\mathbf{L}}_{\mathrm{locality}}}{\underbrace{\gamma \sum _{i,j=1}^n\Vert {\mathbf{z}}_i-{\mathbf{z}}_j{\Vert }_2^2{a}_{ij}}}\end{array}$$

$${\mathbf{L}}_{\mathrm{affinity}}=\frac{1}{2}\Vert \mathbf{Z}-\mathbf{Z}\mathbf{W}{\Vert }_{\mathrm{F}}^2+\beta \Vert \mathbf{W}{\Vert }_{\mathrm{F}}^2.$$

$${\mathbf{L}}_{\mathrm{locality}}=\sum _{i,j=1}^n\Vert {\mathbf{z}}_i-{\mathbf{z}}_j{\Vert }_2^2{a}_{ij}.$$

$$\sum _{i=1}^{l}2c_i(s_i^2{c}_{i-1}+1)-c_1+1+n^2.$$

$$\mathrm{ACC}=\frac{{\sum }_{i=1}^n\delta ({\tau }_i,\mathrm{map}(r_i))}{n}$$

$$\delta (x,y)=\{\begin{array}{ll}1,& \mathrm{if}\text{leavevmode}\text{leavevmode}x=y;\\ 0,& \mathrm{otherwise}\end{array}$$

Full text

1 Introduction

The success of learning algorithms depends highly on feature representation [1] and fully data-driven deep feature learning-based models have better performance than conventional handcrafted feature-based models due to powerful data representation ability of deep models [10]. Meanwhile, unsupervised feature learning is a very important part of deep learning [15] since it is difficult to obtain an amount of high quality labeled data.

Unsupervised feature learning is one of the fundamental topics in the field of machine learning and computer vision. Subspace learning [31], being an especially important branch of unsupervised feature learning, aims to embed the low-level raw data into its latent space. In most subspace methods, each data point is represented by the combination of the whole data set and using the representation coefficients constructs a graph Laplacian for post-processing spectral clustering. Whether features can be well represented to explicitly reflect the data distribution turns out to be a critical factor to the success of unsupervised learning.

Recently, many methods apply deep neural networks to unsupervised learning. Generally, these methods employ deep autoencoder generative model as an initialization, then the learned latent features are applied to different tasks. The loss function used in the fine-tuning stage consists of the network reconstruction error as well as affinity construction error with regularization. However, features obtained from these deep neural networks are directly fed into the fine-tuning stage without further exploiting local pairwise affinity of latent features, which inspires us to exploit well-distributed features with a locally-connected structure to significantly improve the quality of feature learning.

Most existing unsupervised learning methods suffer from some limitations. First, they use inflexible handcrafted features. Second, the representation affinity matrix is learned with shallow methods, such as sparse subspace learning [8], low rank representation [17], spectral curvature clustering [6] etc, which cannot adequately capture the latent data structure. Third, in order to exploit nonlinear functional relations from the raw data space to the latent feature space, the conventional methods use the kernel trick but they still remains a confusion in the choice of kernel function.

With the purpose to tackle the above challenges, as shown in Fig. 1, we frist pretrain a nonlinear generative neural network (GNN) model by stacking convolutional autoencoders, and then we construct an affinity graph with the pretrained GNN latent features to design a new locality-preserving loss function $\bm{L}_{\rm locality}$, and incorporate a self-expression layer in the fine-tuning stage.

In the proposed unsupervised deep local learning (UDLL) method, we focus on two common-sense facts and efficiently take into account the two facts in the well-designed model. Data points in the same cluster have strong connection with high similarity and a data point can be represented by others with coefficients weighing the pairwise affinity, which are the two common-sense facts.

The goal of normalized cut (Ncut) is to partition data points into $k$ weakly inter-connected and strongly intra-connected clusters [28, 22] where $k$ is the class number of objects, and Ncut can effectively reflect and preserve the raw data structure through predefining an affinity graph. Inspired by the great advantage of Ncut, we define a new locality-preserving loss to exploit the local connection information among the latent features. The locality-preserving loss renders latent space features to be of the intra-cluster compactness and inter-cluster separation during the fine-tuning stage. Thus, we use pretrained GNN latent features to construct an affinity graph and optimize with the locality-preserving loss in one integrated network, which can preserve connection structure from the pretrained latent features to the fine-tuned latent features.

Since each latent feature can be represented by other features, such a self-expressive layer is added in the middle of the encoder-decoder generative model and it is fine-tuned to learn a refined affinity between pairwise latent features. The weighted coefficients of the self-expressive layer reflect the pairwise affinity between latent features. With the locality-preserving loss, the structure of the affinity graph is gradually approximate to block-diagonal with reasonable connections during the model iteration.

Comparing to the conventional subspace feature learning methods, the proposed GNN-based deep feature learning method UDLL has following advantages:

Since the handcrafted features can not well preserve important information from raw data, we use deep convolutional neural network features in this paper. 2. 2.

We use pretrained deep GNN features to construct an affinity graph. Using the graph as prior knowledge, the locality-preserving loss is added to the loss function of UDLL. We use the locality-preserving loss besides reconstruction loss of the encoder-decoder GNN model in the fine-tuning stage to preserve local connection structure. 3. 3.

In order to benefit from end-to-end optimization, we add a self-expressive layer in the middle of the encoder-decoder GNN model to learn a new refined affinity graph.

The rest of the paper is organized as follows: we introduce some typical deep neural networks-based feature learning methods in section 2. Then, we introduce our UDLL algorithm in detail in section 3. In section 4, we demonstrate the efficiency of UDLL by conducting plentiful experiments and analyzing the results. Finally, we conclude our work in section 5.

2 Related Work

Unsupervised learning with deep neural networks (DNN) is a relatively new topic. Autoencoders [10, 33] is a typical DNN method to achieve the purpose of feature learning. In most of recent unsupervised DNN-based feature learning algorithms, autoencoders are used as a pretraining procedure to extract hierarchical data features.

Tian et al.[29] use DNN to optimize the reconstruction loss function between the encoder and the decoder, but the input handcrafted feature is firstly optimized by subspace learning and then optimized by DNN secondly. Later, Peng et al. [25] also input handcrafted computer vision datasets for their DNN model resulting in that this method does not effectively utilize the representation ability of a convolutional neural network. As similar as Peng et al.[25]’s method, Ji et al.[12] also make use of the autoencoder as pretraining and self-expressive property to learn the affinity matrix. The subtle difference that Ji et al. [12]’s method inputs raw image data for a convolutional neural network rather than using handcrafted data in the Peng et al.[25]’s model. Taking inspiration from $t$-SNE [19], Xie et al.[35] define an centroid-based auxiliary target distribution to minimize Kullback-Leibler divergence, with parameters initialized by stacked autoencoders. Based on  Xie et al.’s[35] algorithm, Dizaji et al.[7] use cluster assignments frequency as a regularization term to balance the cluster results. Yang et al.[36] jointly optimize a combination of the reconstruction error and $k$-means objective function to achieve ‘clustering-friendly’ latent representations.

Inspired by the fact that deep convolutional neural networks can capture feature in a hierarchical way from a low-level to a high level, Chang et al.[5] adopt the curriculum learning to adaptively select labeled samples for training convolutional neural networks and use a strategy to adaptively choose the label features defined by the cosine similarity. Yang et al.[37] dispose of the successive clustering operations in a recurrent process, stacking the convolutional neural networks representations stepwise. Guo et al.[9] take the data structure into account, employing a clustering loss as prior to prevent the feature space from corruption. Tzoreff et al.[30] lay emphasis on the initial process of deep clustering and propose a discriminative pairwise loss function in terms of the autoencoder pretraining. Based on the popular spectral clustering algorithm, Shaham et al.[27] propose a deep neural network with a constraint in the last layer to satisfy the orthogonality property between the feature vectors.

We systematically compare our method with some of the related work in Table 1 to show the problem we solve. Among all of the above methods, we are the first to construct a graph by the pretraining features as prior knowledge of the fine-tuning stage. The local structure formed in the constructed prior graph is preserved from the pretrained latent feature to the fine-tuned latent feature. In the fine-tuning stage, we refine the latent feature to build a new affinity graph. We add the locality-preserving loss as a structure prior to fine-tune the block-diagonal affinity matrix with higher quality.

3 Unsupervised Deep Local Learning

3.1 Generative Model

Autoencoders are widely used in generative models and typically consist of an encoder and a decoder. As shown in Fig. 1, we adapt convolutional network to form the generative model, the parameters of the encoder are denoted by $\bm{\Theta}_{e}$, and decoder parameters are denoted by $\bm{\Theta}_{d}$. The encoder is denoted by a network $\bm{f}(\bm{\Theta}_{e}):\bm{x}_{i}\rightarrow\bm{z}_{i}$ and the decoder is $\bm{g}(\bm{\Theta}_{d}):\bm{z}_{i}\rightarrow\bm{\hat{x}_{i}}\,$. The loss function of the generative model is defined by the reconstruction cost,

[TABLE]

where $\bm{X}=[\bm{x}_{1},\bm{x}_{2},\ldots,\bm{x}_{n}]$ is the input feature, $\bm{\hat{X}}=[\hat{\bm{x}}_{1},\hat{\bm{x}}_{2},\ldots,\hat{\bm{x}}_{n}]$ is the reconstruction feature, $n$ is the number of data points, and $\|\cdot\|_{\rm F}$ denotes the Frobenius norm.

3.2 Prior Graph Construction

We pretrain the generative model and use the pretrained feature $\bm{Z}=[\bm{z_{1}},\bm{z_{2}},\ldots,\bm{z_{n}}]$ to construct a graph.

Referring to the objective function of the normalized cut [28, 22], we use the following objective to learn the affinity graph ${\bm{A}}=[a_{ij}]$,

[TABLE]

where $\lambda$ is a regularization parameter. By minimizing the above equation, if ${\bm{z}}_{i}$ and ${\bm{z}}_{j}$ have the similar feature, $a_{ij}$ would have a lager value which measure the similarity between them, and vice versa.

In Eq. (2), we constrain ${\bm{A}}\bm{1}=\bm{1}$ so that ${\bm{A}}$ is a normalized graph and its degree matrix $\bm{D}$ is an identity matrix [28] $\bm{D}=\bm{I}$, i.e., $d_{jj}=\bm{1}^{{\top}}{\bm{a}}_{j}=1$ where ${\bm{a}}_{j}$ is the $j$-th column of ${\bm{A}}$. We add the $\ell_{2}$-norm to smooth ${\bm{A}}$ otherwise the solution of Eq. (2) has trivial solution, i.e., only one element is assigned to a value and others are zeroed.

Each column ${\bm{a}}_{j}$ of ${\bm{A}}$ is independent, so we can solve the following problem individually for each $j$:

[TABLE]

When we solve the $j$-th column ${\bm{a}}_{j}$, ${\bm{z}}_{j}$ is a fixed vector with respect to ${\bm{a}}_{j}$. Therefore, we can denote $\|{\bm{z}}_{i}-{\bm{z}}_{j}\|_{2}^{2}$ by a distance metric $m_{ij}$, and solving Eq. (3) is equal to optimizing the problem:

[TABLE]

where ${\bm{m}}=[m_{1j},m_{2j},\ldots,m_{nj}]^{\top}$ is a constant vector.

The Lagrangian function of Eq. (4) is

[TABLE]

where $\eta$ and $\bm{\rho}$ are the Lagrangian multipliers.

According to the Karush-Kuhn-Tucker condition [2], we have following equations,

[TABLE]

Substituting Eq. (10) into Eq. (8), we have

[TABLE]

Substituting Eq. (10) into Eq. (9), we have

[TABLE]

If $\eta^{\star}>\frac{m_{ij}}{2\lambda}\,$, according to Eq. (11), we have $a_{ij}^{\star}\geq\eta^{\star}-\frac{m_{ij}}{2\lambda}>0\,$. When $a_{ij}^{\star}>0\,$, the condition Eq. (12) can only hold if $a_{ij}^{\star}=-\frac{m_{ij}}{2\lambda}+\eta^{\star}\,$.

If $\eta^{\star}\leq\frac{m_{ij}}{2\lambda}\,$, then $a_{ij}^{\star}>0$ is impossible, because it would imply $a_{ij}^{\star}+\frac{m_{ij}}{2\lambda}-\eta^{\star}\geq a_{ij}^{\star}>0\,$, which violates Eq. (12). Therefore, $a_{ij}^{\star}=0$ if $\eta^{\star}\leq\frac{m_{ij}}{2\lambda}\,$.

Thus, we have,

[TABLE]

Without loss of generality, we suppose that $\{m_{1j},$ $m_{2j},$ $\ldots,$ $m_{nj}\}$ are ordered from small to large. If there are only $k$ number of non-zero elements in ${\bm{a}}_{j}^{\star}\,$, then according to Eq. (13), we have $a^{\star}_{kj}>0$ and $a^{\star}_{k+1,j}=0\,$. Accoding to Eq. (7), we have,

[TABLE]

Substituting $\eta^{\star}$ of Eq. (14) into Eq. (13) and considering $a^{\star}_{kj}>0$ and $a^{\star}_{k+1,j}=0\,$, we have

[TABLE]

In order to satisfy to condition in Eq. (15), we set $\lambda$ to,

[TABLE]

According to Eq. (16), we have $2\lambda+\sum_{i=1}^{k}m_{ij}=km_{k+1,j}\,$. Substituting it into Eq. (14), we have $\eta^{\star}=\frac{m_{k+1,j}}{2\lambda}\,$. Thus, using $\eta^{\star}=\frac{m_{k+1,j}}{2\lambda}$ instead of Eq. (13) and considering Eq. (16), the optimal affinity $a^{\star}_{ij}$ is a local $k$-nearest neighbor graph [23],

[TABLE]

The affinity matrix learned by Eq (17) has many advantages: 1) naturally normalized because of $\bm{1}^{\top}{\bm{a}}_{j}=1$; 2) naturally sparse because sparseness is determined by the parameter $k$; and 3) its calculation only involves addition, subtraction, multiplication, and division [23]. Thus, we use it in this paper.

3.3 Unsupervised Feature Learning

As shown in Fig. 1, we add a fully-connected layer called self-expressive layer in the middle of model between encoders and decoders. The UDLL network can achieve a nonlinear map from raw space to latent space, with the self-express layer to further learn a refined affinity matrix. We omit the biases and activations of self-expressive layer, and take ${\bm{Z}}=[{\bm{z}}_{1},{\bm{z}}_{2},\ldots,{\bm{z}}_{n}]$ as input and $\hat{{\bm{Z}}}=[\hat{{\bm{z}}}_{1},\hat{{\bm{z}}}_{2},\ldots,\hat{{\bm{z}}}_{n}]$ as output of the self-expressive layer. In the self-expressive layer, each feature $\hat{{\bm{z}}}_{i}$ can be represented with all other feature ${\bm{z}}_{j}$, i.e., $\hat{{\bm{z}}}_{i}=\sum_{j=1}^{n}w_{ij}{\bm{z}}_{j}$ where $w_{ij}$ is weight of the layer. $w_{ij}$ has a natural meaning to subspace learning which can be thought as a weighting coefficient for representing affinity between $\hat{{\bm{z}}}_{i}$ and ${\bm{z}}_{j}$. With the self-expressive layer, we optimize the following overall loss in the fine-tuning stage,

[TABLE]

where $\alpha$, $\beta$, and $\gamma$ are trade-off parameters.

In the overall loss function Eq. (18), the first term is the reconstruction loss $\bm{L}_{\rm reconstruction}$, the second term is self-expressive loss with regularization $\bm{L}_{\rm affinity}$, and the third term is the locality-preserving loss $\bm{L}_{\rm locality}$ for preserving local structure from pretrained GNN feature space to fine-tuned feature space.

The proposed UDLL model can learn to output an affinity matrix $\bm{W}$ based on self-expressive property. The self-expressive property is inspired by conventional subspace learning methods [8, 17, 24, 34]. By adding self-expressive layer, we can obtain a new refined affinity matrix $\bm{W}$ directly through fine-tuning the whole UDLL model. In this paper, the self-expressive loss with regularization is given by,

[TABLE]

We use the pretrained deep GNN features to construct a prior graph ${\bm{A}}$ which depicts the affinity $a_{ij}$ between pretrained pairwise features ${\bm{z}}_{i}$ and ${\bm{z}}_{j}$. Then, we take into consideration the local connectivity between the pretrained latent data points to design the locality-preserving loss function. Since Ncut [28, 22] defines that vertices with strong connections are partitioned into one component and the weakly connected edges are cut off, inspired by Ncut, the locality-preserving loss is defined by

[TABLE]

Since the prior graph ${\bm{A}}$ is normalized by the constraint ${\bm{A}}\bm{1}=\bm{1}$, Eq. (20) has the same form of original Ncut objective.

The detailed algorithm is summarized in Algorithm 1.

In order to learn the new refined affinity graph $\bm{W}$ by the generative UDLL model, we add a self-expressive layer between encoders and decoders. We take ${\bm{Z}}$ as input to the self-expressive layer, the weights of this layer correspond to the new affinity graph $\bm{W}$. The multiplication ${\bm{Z}}\bm{W}$ represents the newly combined features, and the difference between ${\bm{Z}}$ and ${\bm{Z}}\bm{W}$ measures the self-expressive error. With all of the above, the new affinity graph $\bm{W}$ can be directly solved by Eq. (18).

3.4 Network Architecture

As shown in Fig. 1, the UDLL network consists of three parts: an encoder, a self-expressive layer, and a decoder. The convolutional neural network is employed to build the encoder and the decoder. We use kernels with stride 2 in both horizontal and vertical directions and use rectified linear unit (ReLU) [14] for nonlinear activations. By considering the connectivity of data points, the learned latent representation ${\bm{Z}}$ is more similar to the data points with the same label and more dissimilar to different label points, thus improving the quality of $\bm{W}$.

Suppose that the UDLL network has $l$-layer encoders and decoders with $\{c_{1},c_{2},\ldots,c_{l}\}$ channels. For the $i$-th encoder layer, if the kernel size is $s_{i}\times s_{i}$, the number of weights is $s_{i}^{2}c_{i}c_{i-1}$ with $c_{0}=1$. Due to the symmetric structure of autoencoders, the total number of weights is $\sum_{i=1}^{l}2s_{i}^{2}c_{i}c_{i-1}$ and the number of bias is $\sum_{i=1}^{l}2c_{i}-c_{1}+1$. For the self-expressive layer, the number of $w_{ij}$ is $n^{2}$ when given $n$ number of input raw images. Thus, the total number of parameters of the whole UDLL network is:

[TABLE]

4 Experiments

4.1 Datasets

We use four datasets in our experiment including, COIL-20 [21], COIL-100 [20], ORL [26], and Extended Yale B [16]. The dataset description is summarized in Table. 2. Some sample images of these datasets is shown in Fig. 2.

COIL-20 [21] is from the Columbia object image library and contains 1440 images of 20 objects. Each object contains 72 images. Following Cai et al.[3], images are downsampled to $32\times 32$.

COIL-100 [20] is from the Columbia object image library and contains 7200 images of 100 different objects. Each object contains 72 images. Following Cai et al.[3], images are downsampled to $32\times 32$.

ORL [26] contains 400 images of 40 distinct human faces and each subjects has 10 different images. Following Cai et al.[4], original images are downsampled to $32\times 32$.

Extended Yale B [16] contains 2432 facial images of 38 subjects which is represented by 64 images per subjects. These images are acquired under different illumination conditions. Following Elhamifar et al.[8], images are downsampled to $42\times 42$.

4.2 Network Setting

For different datasets, we use different convolutional neural network architectures.

The UDLL network architecture of COIL-20 consists of one-layer encoders and decoders with 15 channels of kernel size 3$\times$3 and a self-expressive layer with 1440 neurons. In the fine-tuning stage, we take all images as a single batch for training and we set regularization parameters to $\alpha=1000$, $\beta=1$, and $\gamma=19$. The number of the fine-tuning epoch is set to 68. For COIL-20, the number of local nearest neighbors is set to $k=3$ to construct the prior graph calculated by Eq. (17).

For COIL-100, the UDLL network architecture consists of one-layer encoder and decoder with 50 channels of kernel size 5$\times$5 and a self-expressive layer with 7200 neurons. In the fine-tuning stage, we take all images as a single batch for training and we set regularization parameters to $\alpha=15$, $\beta=1$, and $\gamma=280$. The number of the fine-tuning epoch is set to 140. For COIL-100, the number of local nearest neighbors is $k=5$ for the prior graph construction by Eq. (17).

For ORL, the UDLL network consists of three-layer encoder and decoder with $\{5,3,3\}$ channels of kernel sizes $\{5\times 5,3\times 3,3\times 3\}$ and a self-expressive layer with 400 neurons. In the fine-tuning stage, we take all images as a single batch for training and we set regularization parameters to $\alpha=5,\beta=1$, and $\gamma=8$. The number of the fine-tuning epoch is set to 1550. For ORL, the number of local nearest neighbors assigned to each latent data point is $k=3$ for the prior graph construction by Eq. (17).

Extended Yale B is larger than ORL. The UDLL network for Extended Yale B consists of three-layer encoder and decoder with $\{10,20,30\}$ channels of kernel sizes $\{5\times 5,3\times 3,3\times 3\}$ and a self-expressive layer with 2432 neurons. In the fine-tuning stage, we take all images as a single batch for training and we set regularization parameters to $\alpha=3.2$, $\beta=1$, and $\gamma=0.01$. The number of the fine-tuning epoch is set to 1600. For Extended Yale B, the number of local nearest neighbors is $k=10$ for the prior graph construction by Eq. (17).

The Adam optimizer [13] is used to minimize the loss and the learning rate is set to 0.001 in all experiments.

4.3 Experimental Results

The learned features are visualized by $t$-SNE [19] as shown in Fig. 3. It can be seen from Fig. 3 that UDLL separates latent features belonging to different classes very well and UDLL latent features have a distribution that are easily segmented. Fig. 3 implies the impressive representation ability of learned latent features ${\bm{Z}}$ by UDLL. Fig. 3(d) does not well represent the latent feature since we vectorize a tensor concatenated by different channels of feature maps and vectorization renders it to lose two dimensional structure information.

For verifying the representation effectiveness of our UDLL, we test the learned features on clustering task. We evaluate the quantitative results through clustering accuracy (${\rm ACC}$) defined by,

[TABLE]

where total $n$ data points are belonging to $k$ clusters, $\tau_{i}$ denotes the ground-truth label of the $i$-th sample, $r_{i}$ denotes the corresponding learned clustering label, and $\delta(\cdot,\cdot)$ denotes the Dirac delta function

[TABLE]

and map$(r_{i})$ is the optimal mapping function that permutes the obtained labels to match the ground-truth labels. The best mapping is found by the Kuhn-Munkres algorithm [18].

We conduct experiments on four datasets to demonstrate the effectiveness of our feature representation ability of the proposed UDLL algorithm. We compare our algorithm with several baselines:

• •

Low-rank representation (LRR) [17] solved subspace clustering problem by seeking the lowest rank representation among all candidates that can represent the data samples as linear combinations of bases in a given dictionary.

• •

Low-rank subspace clustering (LRSC) [32] posed the subspace clustering problem as a non-convex optimization problem whose solution provides an affinity matrix for spectral clustering, and the goal was to decompose the corrupted data matrix as the sum of a clean and self-expressive dictionary plus a matrix of noise/outliers or gross errors.

• •

Sparse subspace clustering (SSC) [8] aimed to find a sparse representation among the infinitely many possible representations of a data point in terms of other points.

• •

Kernel sparse subspace clustering (KSSC) [24] extended SSC to nonlinear manifolds by using the kernel trick.

• •

Efficient dense subspace clustering (EDSC) [11] dealt with subspace clustering by estimating dense connections between the points lying in the same subspace and formulated subspace clustering as a Frobenius norm minimization problem.

• •

SSC by orthogonal matching pursuit (SSC-OMP) [38] proposed a subspace clustering method based on orthogonal matching pursuit that is computationally efficient and guaranteed to provide the correct clustering.

• •

Deep Embedded Clustering(DEC) conducted unsupervised clustering by first pretraining a deep autoencoder, and then fine-tuning the autoencoder to perform deep embedding learning and clustering jointly.

• •

Deep subspace clustering networks (DSC) [12] was based on a deep autoencoder to find the coefficient representation matrix and applied it to spectral clustering to obtain the clustering results.

The source code of these baselines released by the authors are used and we tune the parameters by grid search to achieve the best results on each datasets.

In the clustering experiments, once we obtain the new affinity graph $\bm{W}$, we can use it to construct an affinity matrix for spectral clustering as same as used in most existing methods. Although affinity matrix $\frac{\bm{W}+\bm{W}^{\top}}{2}$ can be directly fed to spectral clustering, many heuristics have been developed to improve the performance of the constructed affinity matrix. In this paper, we utilize the heuristics used by EDSC [11].

The clustering accuracy of different algorithms on all datasets is provided in Table. 3.

It can be seen from Table. 3 that our UDLL algorithm outperforms all of the state-of-the-art methods, which greatly validate the effectiveness of the locality-preserving loss. From this result, we can make the following conclusions:

• •

The clustering accuracy of DSC and UDLL have a overwhelming advantage over the rest of methods, which can demonstrate the great representation ability of the DNNs as well as the powerful architecture of the autoencoders.

• •

The UDLL outperforms the DSC in terms of accuracy on all of the four datasets, implying the good quality of the refined affinity matrix for more accurate clustering result.

• •

The locality-preserving loss can well capture the local connections from pretrained GNN feature space to the fine-tuned stage feature space, improving the representation ability of affinity matrix.

• •

We formulate our novel local-preserving loss under a strong theoretical foundation of Ncut, and the intra-class compactness and inter-class separation is the key to improve the clustering accuracy.

4.4 Ablation Study

To further evaluate the feature representation ability of UDLL and the effectiveness of the structure connection constraint locality-preserving loss, we directly apply EDSC to the pretrained graph and the fine-tuned graph, respectively. As shown in Fig. 4, the fine-tuned feature has better performance than the pretrained feature.

Comparing with the UDLL clustering results with fine-tuned features, we show the different results using pretrained features with SSC (PT$+$SSC) and EDSC (PT$+$EDSC) in Table 4. It can be seen from Table 4 that the fine-tuned model obtains the better results than others.

5 Conclusion

We proposed a new algorithm called UDLL. UDLL is a generative model with an additional self-expressive layer. The self-expressive layer is used to compute the coefficient matrix and the matrix is used to construct an affinity matrix for post-processing spectral clustering. First, we pretrained autoencoders and obtain the latent representation of input data. Second, the latent representation was used to construct a prior graph which describes the affinity between pairwise latent features. Third, we fine-tuned the UDLL model with a self-expressive layer and with connectivity regularization by the prior graph. The prior graph was constructed by a $k$-nearest neighbors algorithm. The prior graph formed the data structure of pretrained latent feature which was preserved from pretraining to fine-tuning by optimizing the locality-preserving loss. By considering the connectivity constraint in the prior graph, experiments on four image datasets had demonstrated that UDLL feature has a powerful representation ability and UDLL provided a significant improvement over state-of-the-art methods.