Deep Representation Learning Characterized by Inter-class Separation for Image Clustering

TL;DR

This paper introduces a deep autoencoder-based method that automatically learns discriminative features for image clustering, improving accuracy and generalization without relying on pre-trained models.

Contribution

It proposes a novel deep autoencoder approach that simultaneously learns features and discriminates between similar and dissimilar images for better clustering.

Findings

Outperforms state-of-the-art clustering methods

Works well across different datasets without pre-trained models

Produces more meaningful and separated clusters

Abstract

Despite significant advances in clustering methods in recent years, the outcome of clustering of a natural image dataset is still unsatisfactory due to two important drawbacks. Firstly, clustering of images needs a good feature representation of an image and secondly, we need a robust method which can discriminate these features for making them belonging to different clusters such that intra-class variance is less and inter-class variance is high. Often these two aspects are dealt with independently and thus the features are not sufficient enough to partition the data meaningfully. In this paper, we propose a method where we discover these features required for the separation of the images using deep autoencoder. Our method learns the image representation features automatically for the purpose of clustering and also select a coherent image and an incoherent image simultaneously for a given image so that the feature representation learning can learn better discriminative features for grouping the similar images in a cluster and at the same time separating the dissimilar images across clusters. Experiment results show that our method produces significantly better result than the state-of-the-art methods and we also show that our method is more generalized across different dataset without using any pre-trained model like other existing methods.

Figures

Tables

Table 1. Table 1: Clustering performance based on accuracy (%), given by the Equation 8 , of different algorithms(higher is better). We report the mean of 20 runs. † † \dagger denotes using weights of pre-trained deep residual networks [ 13 ] . Results marked with * are excerpted from [DEPICT]. We put dash marks (-) where method does not provide any result for the dataset.

	MNIST	USPS	CIFAR10	CIFAR100	FRGC	ImageNet
$K$-means on pixels	53.49	46	20.4	16.5	24.3	12.65
DEC [34]	84	62*	-	-	37.8	-
VaDE [39]	94.46	-	-	-	-	-
JULE [36]	96.4	95*	-	-	46.1	-
DEPICT [8]	96.5	96.4	-	-	47	-
IMSAT [16]	98.4	-	45.6$†$	27.5$†$	-	-
Ours	98.93	97.63	44.19	25.4	47.28	35.69

Table 2. Table 2 : Clustering accuracy (%), given by the Equation 8 , on an imbalance sub-sample of MNIST.

	0.1	0.3	0.5	0.7	0.9
$K$-means on pixels	46.96	48.73	52.86	53.16	53.39
DEC	70.10	80.92	82.68	84.69	85.41
Ours	88.02	94.5	96.12	97.2	97.91

Equations

$$L_r=\frac{1}{n}\sum _{i=1}^n||G_{\theta }(F_{\varphi }(x_i))-x_i|{|}_2^2$$

$$q_{ij}=\frac{(1+|z_i-{\mu }_j{|}^2{)}^{-1}}{{\sum }_j(1+|z_i-{\mu }_j{|}^2{)}^{-1}}$$

$$p_{ij}=\frac{q_{ij}^2/{\sum }_i{q}_{ij}}{{\sum }_j{q}_{ij}^2/{\sum }_i{q}_{ij}}$$

$$L_c=KL(P||Q)=\sum _i\sum _j{p}_{ij}log\frac{p_{ij}}{q_{ij}}$$

$$L_t=\sum _{i=1}^n\max (0,[||z_i^a-z_i^p|{|}_2^2-||z_i^a-z_i^n|{|}_2^2+margin])$$

$$\begin{array}{r}{L}_b=(\sum _i\sum _{j=1}^K{q}_{ij}log(q_{ij}))-(\sum _{j=1}^K\overline{q_{ij}}log(\overline{q_{ij}}))\\ \text{where }\overline{q_{ij}}=\frac{1}{B}\sum _i^B{q}_{ij}\end{array}$$

$$L=\alpha L_r+\beta L_c+\gamma L_b+\omega L_t$$

$$ACC=ma{x}_m\frac{{\sum }_{i=1}^{n}1\{l_i==m(c_i)\}}{n}$$

Full text

1 Introduction

Clustering of images is one of the fundamental and challenging problems in computer vision and machine learning. Several applications like 3D reconstruction from an image set [9], storyline reconstruction from photo streams [19, 20], web-scale fast image clustering [2, 12] etc. use image clustering as one of the important tools in their methodology. While the literature for clustering algorithm is voluminous [29, 28, 38, 37, 33, 36, 10], there are roughly two types of approaches for clustering viz. hierarchical clustering and centroid-based clustering [14]. Most popular algorithm in the hierarchical methods is agglomerative clustering which begins with many small clusters, and then merges clusters gradually [10, 36] to produce a certain number of clusters. On the other hand, centroid based methods (e.g. $K$-means ) [38] pick $K$ samples from the input data for the initialization of the cluster centroids which are further refined iteratively by minimizing the distance between the input data and the centroids. All of these methods require the notion of feature to represent an image data so that a meaningful partition of an image dataset can be obtained. If the dimension of the feature is very high then it becomes ineffective to compute the distance between the features due to the well-known fact of curse of dimensionality [35]. Hence a variety of methods like principal component analysis (PCA), canonical correlation analysis (CCA), nonnegative matrix factorization (NMF) and sparse coding (dictionary learning) etc. are regularly used to reduce the dimensionality of features before clustering. Therefore the performance of the clustering methods crucially depends on the choice of these different feature representations.

In recent years, deep-learning (DL) methods have made an enormous success in producing good feature representation of an image which is required for different machine learning tasks [13, 23, 6]. DL learns these powerful representations from the image through high-level non-linear mapping [3] and hence these representations can be used to partition the data into different clusters. $K$-means works better with such representations learned by DL models than using other traditional methods [35]. There are broadly two approaches to use the deep representation of the data for clustering. The first naive approach is to use the hidden representation (features) of the data which are extracted from a well-trained deep network [31] using supervision. However, this approach cannot fully exploit the power of deep neural network for unsupervised clustering because the usage of an already trained deep network for some other purpose lacks of knowledge of feature required for partitioning an unknown data. In the second approach an existing clustering method is embedded into a DL model. This association enables the DL model to learn cluster-oriented meaningful representation. For example, Song et al. [30] integrate $K$-means algorithm into a deep autoencoder and does cluster assignment in the bottleneck layer. Such an approach produces better outcome as the features learned by the DL models are relevant for clustering. Also, if we can predict the distribution of the cluster assignment of the features through some auxiliary distribution then, the observed cluster assignment of a feature and the auxiliary distribution together can improve the accuracy of the clustering. To this end, in a recent work, Xie et al. [34] propose a clustering objective to learn deep representation by minimizing the KL divergence between the observed cluster assignment probability and an auxiliary target distribution which is derived from current soft cluster assignment. For designing the auxiliary distribution Xie et al. [34] use a function of current model assignment distribution and the frequency per cluster. However, the cluster assignment probabilities keep on changing with training and therefore without any constraints, the auxiliary distribution also keeps on changing resulting in poor quality of representation as shown in Figure 1(a). To improve the representation, in a recent method Hu et al. [16] propose a new regularization on their DL model which enforces that feature representations of intra-class should come close to each other to improve the representation learning. They use a pre-trained model for feature enrich datasets like CIFAR10 and CIFAR100 but do not achieve comparable accuracy with other datasets like MNIST. Along with the constraints like feature coherence of images for same class, clustering methods should also have an important constraint like the separation between inter-class features should also be high. But in an unsupervised setting detection of image which does not belong to same class is difficult.

In this work, we use a new constraint on Deep Convolutional Autoencoder (DCA) model whose objective to bring intra-class images closer and make inter-class images distant in their representation space simultaneously. To this end, for every anchor image in the database we select a positive image which is an augmented version of the anchor image and a negative image which is neither similar to the original anchor image nor the augmented image in their latent representation space. The negative image is selected from the image database using our proposed selection algorithm described in Section 3.3. We put a constraint in clustering using this image triplet (anchor image, positive image and negative image) to minimize the distance between the anchor and the positive image and at the same time maximize the distance between the anchor and the negative image in their latent representation space. This new constraint allows the model to learn better feature representation which in turn produce a better auxiliary distribution (as this is a function of the feature representation [34]) required for improved accuracy in clustering being in unsupervised setting. We also propose a method to avoid a degenerate condition where an image belonging to a class can have a considerable probability to belong to multiple clusters during training due to the unconstrained model assignment probability. To mitigate this, we use the difference between the entropy of the average model assignments and the entropy of model assignments inspired from  [5] as an additional constraint, which enforces to produce unambiguous model assignments.

The main contributions of this paper are :

• •

We use a new constraint on deep representation learning to enforce that feature representation can learn both intra-class similarities and inter-class separation at the same time in an unsupervised setting which helps to improve clustering.

• •

Our method use entropy minimization for model assignment probability which helps the model to produce better model assignments.

• •

In contrast to some other recent methods which use features from some already pre-trained models for feature rich dataset like CIFAR10, our method does not depend on such existing pre-trained models. Also our method shows excellent performance in all comparative dataset which demonstrates the generalization of our method in terms of applicability.

The rest of the paper is divided into four sections, Section 2 present the background study on unsupervised image clustering. Section 3 describes of our method consisting of our clustering network, inter-class separation constraint and selection algorithm. We present our experiments in Section 4 followed by a conclusion.

2 Related Work

Classical clustering algorithms such as $K$-means [26], Gaus sian mixture models [4] and spectral clustering [33], work with original data spaces and are not very efficient in high dimensional spaces with complicated data distribution such as images [35].

Early attempts in deep learning based clustering approach first train the feature representation model and then in a separate step they use these learned features for clustering [31]. In recent years, the representation learning and clustering is done based on joint optimization on deep network [34, 39, 8, 16, 36]. Using this joint optimization method, Yang et al. [36] propose a new clustering model Joint Unsupervised Learning (JULE), which learns representation and clustering simultaneously. It uses agglomerative clustering and achieves a good result, but it requires high memory and computational power [15]. Deep Embedded Clustering (DEC) [34] model simultaneously learns feature representations using an autoencoder along with cluster assignments. The encoder is pre-trained layer-wise and then fine-tuned by a clustering algorithm using KL divergence. The problem with this approach is that their representation does not adequately address inter-class separation and hence not scalable for the high dimensional dataset. Variational Deep Embedding (VaDE) [39] is a generative clustering approach using variational autoencoders. It uses variational autoencoders and Gaussian mixture model simultaneously to model the data generative process. This method also does not generalize well as it requires a pre-trained model for high dimensional image datasets. Deep Embedded Regularized Clustering (DEPICT) [8] uses joint optimization to learn feature representation and cluster assignment using a convolutional autoencoder. It uses a regularization, which enforces that each cluster has an almost equal number of samples. But this leads to a bad clustering performance when data is not uniformly distributed across clusters, or the prior data distribution is unknown. Most recent deep learning based model is Information-Maximizing Self-Augmented Training (IMSAT) [16]. It is based on Regularized Information Maximization (RIM) [22] which maximizes the mutual information between the input and the cluster assignment by learning a probabilistic classifier. It does augmentation to put a new constraint in its objective function where it tries to minimize the $L2$ distance between augmented data and original data in their latent representation space to understand the intra-class similarities of samples. It also requires some labeled data for augmentation and uses fixed pre-trained network layers for feature enrich dataset like CIFAR10 [32] and CIFAR100 [32].

So, there are broadly two types of problems with the previous works (1) unable to learn deep representations which are characterized by both intra-class similarities and inter-class dissimilarities [34, 8, 16] , (2) issue of prior knowledge of data distribution [8]. In this paper, we propose new constraints for both representation learning and clustering to solve these problems.

3 Proposed Methodology

In this section, we first introduce our convolution autoencoder and the clustering layer of our clustering network. Then, we describe in detail about the clustering constraints required for our network and their formulation. Finally, we describe the composite objective function for our network which simultaneously estimates the cluster assignment and updates the network parameters.

3.1 Deep Convolutional Autoencoder

Our deep convolutional autoencoder (DCA) has an encoder layer which maps the input image $x_{i}$ to it’s latent representation $z_{i}=F_{\phi}(x_{i})$ using a stack of convolution layers followed by a fully connected layer. The decoder layer has a fully connected layer followed by a stack of deconvolutional layers to convert the latent representation back to original image $x_{i}^{\prime}=G_{\theta}(z_{i})=G_{\theta}(F_{\phi}(x_{i}))$ where $\phi$ and $\theta$ are the parameters of the encoder and the decoder respectively. The corresponding architecture is shown in Figure 2. We minimize reconstruction loss given by the Equation 1. The purpose of using the reconstruction loss is to preserve the local structure of the input images into features space.

[TABLE]

3.2 Clustering Layer

The clustering layer in our network consists of cluster centers $\mu_{j}$ where $j={1....K}$, where $K$ is the number of predefined clusters. Our clustering layer is similar to the network which is proposed by [34]. Similar to [25] we use student-t distribution to measure the similarity between $z_{i}$ and the cluster center $\mu_{j}$. The probability (model assignment) of assigning sample $i$ to cluster $j$ is given by the Equation 2.

[TABLE]

In absence of the target label in an unsupervised setup, we use the auxiliary probability from [34] for assigning a data $i$ to a cluster $j$ as given by Equation 3.

[TABLE]

Now, we compute the clustering loss $L_{c}$ using the KL divergence between model assignment probability distribution and the auxiliary distribution as given by the Equation 4.

[TABLE]

KL divergence loss given by the Equation 4, cannot produce a better discriminative representation in an unsupervised setting where actual target distribution is unavailable. Figure 1(a) shows that there exist an overlapping distribution of latent representations for class 1 and 7 of MNIST. This overlapping representation occurs due to change in auxiliary probability distribution (Equation 3) as training progress. To mitigate this problem, we use a new constraint on representation learning to learn more discriminating feature by simultaneously minimizing the $L2$ distance between images of the same class and maximize the $L2$ distance among inter-classes.

3.3 Representation Learning Using Inter-class Separation Constraint

We use a new constraint, given by the Equation 5, whose objective is to minimize the $L2$ distance between similar images and maximize the $L2$ distance among inter-classes in their featrure representation space. This constraint enforces that the features of the anchor image ($I_{a}$) and the positive image ($I_{p}$) should come close and at the same time the distance between the features of the anchor image ($I_{a}$) and the negative image ($I_{n}$) should increase.

[TABLE]

• •

$z_{i}^{a}$ is the representation of the anchor image, $I_{a}$, i.e $z_{i}^{a}=F_{\phi}(I_{a})$

• •

$z_{i}^{p}$ is the representation of the positive image, $I_{p}$, i.e $z_{i}^{p}=F_{\phi}(I_{p})$

• •

$z_{i}^{n}$ is the representation of the negative image, $I_{n}$, i.e $z_{i}^{n}=F_{\phi}(I_{n})$

• •

$margin$ ensure the network doesn’t optimize towards $z_{i}^{a}$ - $z_{i}^{p}$ = $z_{i}^{a}$ - $z_{i}^{n}$ = 0.

We use different augmentation methods like rotation, scale, erasing, flip, to produce the positive image ($I_{p}$) from the anchor image ($I_{a}$). It is a difficult task to select a negative image ($I_{n}$) for the anchor image ($I_{a}$) in an unsupervised setting where labels are unknown. Random selection of a negative image ($I_{n}$) from the image dataset cannot guarantee that it is truly negative with respect to the anchor image ($I_{a}$) and therefore may fall into the same class that of the anchor image ($I_{a}$). So, we propose an algorithm to select the proper negative image ($I_{n}$) to enforce that negative image ($I_{n}$) does not belong to the same class of the anchor image ($I_{a}$) to satisfy the necessary condition for the Equation 5.

Our selection algorithm tries to discover the difference between the distribution of the augmented images and the anchor image in their latent representation space by using $L2$ distance. It selects the appropriate image as a negative image from the dataset whose latent representation does not fall within the distribution of the augmented images of the anchor image. Algorithm (1) describes the complete algorithm for selecting a negative image. We pre-trained our DCA for three epochs using reconstruction loss to initialize the parameter $\phi$ of the encoder to enable our Algorithm (1) to use the latent representation for selecting negative images.

Figure 3 demonstrates our selection process. Such a method enables the network to learn better representation which helps in improvement clustering accuracy. However, an image belonging to a class can have a considerable probability to belong to multiple clusters during training due to the unconstrained model assignment probability over classes as mentioned in Equation 2. For instance, Figure 4(a) shows that for MNIST dataset, $35$% of the images from class $1$ have high probabilities for belonging to its own class and rest of the samples from class $1$ belongs to other classes during training after $10$ epoch. This leads to an incorrect clustering without explicit constraint as shown in our ablation study in Figure 5(a). To mitigate this, we use a constraint, given by the Equation 6, based on the difference between the entropy of the average of the $q_{ij}$ (Equation 2), and the entropy of the $q_{ij}$ (Equation 2) so that we can retain the information of the input in $q_{ij}$ similar to  [5]. Figure 4(b) shows the histogram of the images from class $1$ from MNIST after adding this constraint (Equation 6) which clearly indicate that the class assignment becomes significantly better during training resulting in improved accuracy overall. The corresponding ablation study is shown in Figure 5(a).

[TABLE]

Here $B$ is the batch size and $q_{ij}$ is taken from the Equation  2.

3.4 Objective Function

The final loss function for our network given by the Equation 7.

[TABLE]

with the hyper parameter of $\alpha$ , $\beta$ , $\gamma$ , $\omega$. We use all these constraints to optimize our network. We show in our ablation study the efficacy of each of the terms.

4 Experiments

4.1 Datasets

We evaluate our proposed clustering network on the following, widely-used image datasets:

• •

The MNIST dataset [24] consists of total 70000 handwritten digits of $28\times 28$ pixels. We use all images from the training and test sets without their labels.

• •

USPS: It is a handwritten digits dataset from the USPS postal service [17], containing 11,000 samples of $16\times 16$ images.

• •

CIFAR-10 [32] contains $32\times 32$ color images of ten different object classes. Here, we use all images of the training set.

• •

CIFAR-100 [32] contains $32\times 32$ color images of hundred different object classes. Here, we use all images of the training set.

• •

FRGC [36]: We use the method proposed by [36] to select 20 random subjects from the original dataset and collect 2,462 face images. We crop the face regions and resize them into $32\times 32$ images.

• •

ImageNet [7]: We select 10 random classes from the original dataset and collect 13000 images. We crop and resize them into $32\times 32$ images.

4.2 Evaluation Metric

The clustering method is evaluated by the unsupervised clustering accuracy(ACC) [34].

[TABLE]

where $l_{i}$ is is the ground-truth label, $c_{i}$ is the cluster assignment i.e $c_{i}$ = $argmax_{j}$($q_{ij}$) and $m\in M$ are all possible one-to-one mappings between clusters and labels.

4.3 Experiment Result

We use Tensorflow [1] to implement our method on a Linux based workstation equipped with GPU. We report our result based on two types of networks, viz a shallow network suitable for the datasets like MNIST, USPS and a deeper variant of the shallow network with more parameters for the datasets like CIFAR10, CIFAR100, FRGC and ImageNet which have more features than MNIST or USPS. The encoder of the shallow network consists of three convolutional layers, with kernel size $3\times 3$ and stride $1$, followed by a fully connected layer. All the convolutional layers associate with a max-pooling layer with stride $2$. The decoder of the shallow network consists of a fully connected layer followed by three deconvolutional layers. The deeper variant of our shallow network uses additional two convolution and two deconvolution layers for the encoder and the decoder respectively. For both the networks we apply a zero padding for all the convolutional and max-pooling layers. We use rectified linear units [27] for all the hidden activations and apply batch normalization [18] to each layer. For initialization of weights of all the layers we choose the Xavier approach [11]. We select Adam [21] optimizer for the back propagation with learning-rate of $0.001$. The feature dimensions of our shallow and deep network are $64$ and $10$ respectively. We normalize the input mages to zero-mean and unit variance and pre-trained the network with only reconstruction loss to initialize the cluster centers using $K$-means.

Table 1 shows the complete results for all the datasets using our method. Our method produces significantly better accuracy on MNIST, USPS, and FRGC. We have achieved the competitive result on CIFAR10 and CIFAR100. IMSAT gives the best accuracy on CIFAR10 and CIFAR100 by using the features of an already pre-trained model of deep residual networks [13]. This deep residual network [13] is trained on ImageNet dataset of natural images which is very similar to CIFAR (dataset of natural images). Features extracts for CIFAR10 and CIFAR100 from this model have good representation which is very helpful for clustering but they do not show any result on ImageNet itself. We use the same pre-trained model on FRGC for the clustering and have obtained very poor accuracy (24.31%) which shows that use of a pre-trained model in a method do not generalize well. In contrast, we use our network for FRGC, CIFAR10, CIFAR100 and ImageNet. We exceed state-of-the-art accuracy on FRGC and comparable accuracy with respect to IMSAT on CIFAR. To the best of our knowledge we are the first to try out ImageNet dataset for clustering. We show that in general, our method produces significantly better results with more generalization across datasets. However, there is a substantial difference between results on the datasets like MNIST, USPS and the feature rich datasets (CIFAR10, CIFAR100, FRGC, ImageNet). This difference is due to the fact that the auxiliary probability distribution is not very close to the true distribution of data which impacts the accuracy of the feature rich dataset.

4.4 Ablation Study

To understand the importance of each constraint in our method, we train and evaluate a series of models, each eliminating one or more constraints from Equation 7. We choose MNIST and CIFAR10 datasets for the ablation study. The results of our ablation study shown in Figure 5. It is evident from this study that the inter-class separation constraint ($L_{t}$) has a significant influence in achieving the improved accuracy. We also conduct an extensive experiment on MNIST to find the suitable values for the hyper parameters used in Equation 7. We sample the hyper parameters in a range of $[0;1]$ and experiment on the shallow network for $100$ times. We compute the accuracy based on the evaluation metric given by the Equation 8. We keep one hyper parameter $\alpha$ fixed as $1$ always and change the rest of the hyper parameters as shown in Figure 6. Based on this experiment we choose hyper parameters for MNIST and USPS as $\alpha=1$ , $\beta=0.8$ , $\gamma=0.01$ , $\omega=0.1$. We conduct the same experiment for deeper version of the network for CIFAR10, CIFAR100, FRGC, ImageNet and find the hyper parameters as $\alpha=1$ , $\beta=0.7$ , $\gamma=0.01$ , $\omega=0.3$. We set $margin$, used in Equation 5, as 0.24 and 0.3, for shallow and deeper network respectively.

4.5 Effect of Imbalanced Data

In order to check the performance in an imbalance dataset, we use a similar approach proposed by [34] for MNIST dataset. We use retention rate $r$ to keep the sample for a given class of MNIST. For example, if minimum retention rate is $r_{min}$, then it implies that we randomly select samples from class 0 with the probability of $r_{min}$ and class 9 with the probability of 1, and samples from other classes fall linearly in between $r_{min}$ and 1. All the recent methods, mentioned in the Table 1, do not provide any result on an imbalance dataset except DEC. Table 2 shows that our method produces significantly improved accuracy than DEC [34] for an imbalance dataset as well.

5 Conclusion

In this work, we propose a deep learning based clustering method which learns the high intra-class similarity and low inter-class similarity simultaneously in the latent representation space. We carefully design our selection algorithm to choose a negative image for an anchor image in an unsupervised setting. Owing to our careful choices of the constraints, the model assignment probability in our method is improved which in turn improve the auxiliary distribution resulting in improved accuracy in clustering. Improving the auxiliary distribution further through a separate learning strategy can be an immediate future work of our method for even better clustering for complex dataset like ImageNet.