One-Class Feature Learning Using Intra-Class Splitting

TL;DR

This paper introduces a new one-class feature learning approach that splits normal class data into typical and atypical samples, enabling effective feature extraction without multi-class pretraining, and demonstrates superior performance on image datasets.

Contribution

The proposed intra-class splitting method allows for effective feature learning in one-class classification without relying on multi-class pretraining, a novel approach in the field.

Findings

Outperformed baseline models on three image datasets

Effective in extracting valuable features for one-class classification

Demonstrated robustness across different datasets

Abstract

This paper proposes a novel generic one-class feature learning method based on intra-class splitting. In one-class classification, feature learning is challenging, because only samples of one class are available during training. Hence, state-of-the-art methods require reference multi-class datasets to pretrain feature extractors. In contrast, the proposed method realizes feature learning by splitting the given normal class into typical and atypical normal samples. By introducing closeness loss and dispersion loss, an intra-class joint training procedure between the two subsets after splitting enables the extraction of valuable features for one-class classification. Various experiments on three well-known image classification datasets demonstrate the effectiveness of our method which outperformed other baseline models in average.

Figures

Tables

Table 1. TABLE I: Balanced Accuracy in %.

Dataset	HOG	PCA	ImageNet	CAE	Original	CLS	Ours
MNIST	64.3 $\pm$0.0	75.2 $\pm$0.0	68.7 $\pm$0.0	85.2 $\pm$0.7	84.2 $\pm$0.0	84.7 $\pm$5.6	91.3 $\pm$0.7
FMNIST	81.1 $\pm$0.0	81.7 $\pm$0.0	65.2 $\pm$0.0	81.6 $\pm$1.0	82.3 $\pm$0.0	84.8 $\pm$2.5	87.7 $\pm$0.5
CIFAR-10	53.8 $\pm$0.0	57.1 $\pm$0.0	53.7 $\pm$0.0	56.9 $\pm$1.4	56.5 $\pm$0.0	54.6 $\pm$3.0	60.6 $\pm$1.2

Equations

$${\mathcal{L}}_{\mathrm{rec}}+\alpha {\mathcal{L}}_{\mathrm{cls}}+{\beta }_1{\mathcal{L}}_{\mathrm{disp},1}+{\beta }_2{\mathcal{L}}_{\mathrm{disp},2},$$

$${\mathcal{L}}_{\mathrm{rec}}=\frac{1}{B}\sum _{j=1}^B||{\mathbf{x}}_j-{\hat{\mathbf{x}}}_j|{|}^2,$$

$${\mathcal{L}}_{\mathrm{cls}}=\frac{1}{B}\sum _{j=1}^B\sqrt{\frac{1}{L}||{\mathbf{z}}_j-{\mathbf{z}}_{i\ne j}|{|}^2},$$

$$\begin{split}\mathcal{L}_{\mathrm{disp}}=&\beta_{1}\cdot\underbrace{\Big{(}-\frac{1}{B}\sum_{j=1}^{B}\sqrt{\frac{1}{L}||\bm{z}_{j,\mathrm{atypical}}-\bm{z}_{i\neq j,\mathrm{atypical}}||^{2}}\ \Big{)}}_{\mathcal{L}_{\mathrm{disp},1}}\\ &+\beta_{2}\cdot\underbrace{\Big{(}-\frac{1}{B}\sum_{j=1}^{B}\sqrt{\frac{1}{L}||\bm{z}_{j,\mathrm{atypical}}-\bm{z}_{j,\mathrm{typical}}||^{2}}\ \Big{)}}_{\mathcal{L}_{\mathrm{disp},2}}~{},\end{split}$$

$$\mathrm{BACC}=\frac{1}{2}\Big{(}\frac{\mathrm{TP}}{\mathrm{TP}+\mathrm{FN}}+\frac{\mathrm{TN}}{\mathrm{TN}+\mathrm{FP}}\Big{)}~{}.$$

Full text

I Introduction

One-class classification is a subfield in machine learning, aiming at identifying normal data from abnormal data using a training dataset consisting merely of samples from the normal class. It is more challenging than binary or multi-class classification in which samples from all classes are available during training. Various one-class classifiers were proposed and successfully applied to a wide range of applications including fault detection, novelty detection or anomaly detection [1, 2]. Schölkopf et al. [3] or Tax et al. [4] proposed state-of-the-art one-class classifiers which are characterized by a tight decision boundary around the training samples of the normal class in order to reject abnormal samples of different kinds during inference.

To achieve a tight decision boundary, one-class classifiers require an input feature space that fulfills the following two conditions. First, features of normal data must be compactly distributed. We call this requirement closeness. Second, normal and abnormal data must have large distances between each other in the feature space. This requirement is called dispersion. Both requirements are comparable to the linear discriminant analysis in clustering in which the sample distances within a cluster are minimized and the sample distances between clusters are maximized [5].

As the stated conditions are typically not fulfilled for high-dimensional data such as natural images, the performance of state-of-the-art one-class classifiers is quite limited. Hence, a feature extraction method is necessary for such classifiers to transform raw data into a suitable latent feature space satisfying the closeness and dispersion requirements. However, this is challenging, because abnormal data samples are not available during training. Therefore, few work was done on feature learning for one-class classification.

State-of-the-art feature learning methods for one-class classification are supported by samples from other classes. In particular, a reference dataset is used to pretrain a model on many other classes [6]. The underlying assumption is that real abnormal samples might be contained in these classes. However, this assumption is too strong, since the number of other classes is unlimited. If the reference dataset is not representative for real abnormal classes, then the decision boundary of a one-class classifier will be too loose. Therefore, the choice of a reference dataset is crucial. Indeed, a meaningful multi-class reference dataset is typically not available due to the nature of the one-class classification problem.

In this paper, our goal is to learn a latent space that fulfills both closeness and dispersion requirements using only normal data. As a result, latent representations are optimized to the training class and not biased to any outliers. A solution for this goal does not only enable the use of conventional one-class classifiers to the learned feature space, but has also high potential for unsupervised learning and open set recognition [7]. Intuitively, we assume that in a certain latent space, normal data is clustered and abnormal data distributes around it. Fig. 1 illustrates this scenario.

In order to find such a latent space, our first idea is to split a normal training set into two subsets in an unsupervised manner and by using a similarity metric. One subset includes typical normal data, meaning the majority of a given training dataset, and the other subset consists of atypical normal data, meaning the minority of the training set. Hence, no reference dataset is necessary in contrast to [6]. By using an joint training procedure to maximize the dispersion between the typical and atypical normal data, a tight decision boundary around the normal data in the latent feature space can be achieved. This is our second key idea.

One meaningful similarity metric for image data is the structural similarity (SSIM) [8]. For example, Fig. 2 illustrates the division of exemplary classes into typical and atypical normal samples using SSIM. In more detail, an autoencoder is first trained on the whole training set. Subsequently, the similarity metric between reconstructed and original data is calculated. Finally, the data with higher similarity is considered to be typical, whereas the samples with lower similarity are considered to be atypical normal data.

In general, this work is based on the assumption that unknown abnormal data has more common features with the atypical normal data rather than typical normal data in a certain latent space. In particular, we use the atypical normal samples from the given training dataset to model the unknown abnormal classes. This assumption is weaker than those in prior work such as [6], since we do not restrict the unseen abnormal data to a limited number of reference datasets, which may lead to a too loose decision boundary and a subsequent poor sensitivity. Instead, only the intra-class information from the normal class is utilized and a tight decision boundary is thus expected.

In summary, our main contributions in this paper are the following:

• •

Intra-class splitting is firstly introduced which is the key to solve feature learning for one-class classification. However, it is a generic method and not limited to one-class feature learning.

• •

A novel intra-class joint training strategy: Both typical and atypical normal subsets have their individual objectives as well as common objectives. These objectives are finally reformulated as a joint optimization problem.

• •

We empirically prove that closeness plays a key role in feature learning for one-class problems. Furthermore, it is shown that the combination of closeness and dispersion can achieve an even better feature extraction performance.

II Proposed Method

II-A Overview

As mentioned in the introduction, the goal is to learn a model which extracts features based on one class only, fulfilling both closeness and dispersion in the latent space. Therefore, the basic idea is to split the given normal data into typical and atypical normal samples and to use a three-stage joint training procedure. First, the foundations and strategy of intra-class splitting are explained. Subsequently, three desired characteristics of the latent space are briefly described. Then, the corresponding loss functions are introduced. Finally, an autoencoder-based network considering intra-class dispersion and closeness is presented.

II-B Intra-Class Splitting

In a given normal class, not all samples are representative for this class as illustrated in Fig. 2. Accordingly, it is assumed that a given normal dataset is composed of two parts, typical normal samples and atypical normal samples. Typical normal samples are the most representative for the normal class and correspond to the majority of the given dataset. In contrast, the remaining samples are considered as atypical normal samples which may mislead the learning of a one-class classifier and thus are used to model the abnormal classes.

Intuitively, an approach to realize the splitting is utilizing neural networks with a bottleneck structure. By using a compression-decompression process such as in an autoencoder, input data is first transformed into a low-dimensional representation with information loss and then mapped back to the original data space. Hence, only the most important information of the given dataset is well maintained during this process. Accordingly, the samples contain more representative features if they are better reconstructed.

Formally, a given normal dataset $\chi$ is split by using a predefined similarity metric $f(\bm{x},\hat{\bm{x}})$ and a ratio $\rho$ where $\hat{\bm{x}}$ is the reconstruction of a sample $\bm{x}\in\chi$. In particular, the first $\rho\%$ samples with the lowest similarity scores are considered as atypical normal samples $\chi_{\mathrm{atypical}}$, while the others are considered as typical normal samples $\chi_{\mathrm{typical}}$.

II-C Desired Characteristics of the Latent Space

II-C1 Closeness

Input data typically distributes along a manifold in the original high-dimensional space. Therefore, it should naturally distribute in a small region in a low-dimensional latent space. In this low-dimensional space, all latent representations of normal data should be as close to each other as possible. In other words, the intra-class latent representations of normal data must have a high closeness among themselves.

II-C2 Dispersion

While closeness is necessary for intra-class latent representations of normal data, abnormal data must be as far away from normal data as possible in the latent space. Hence, abnormal data should have a high dispersion comparing with normal data in this space. As mentioned in the introduction, a training set is assumed to consist of typical and atypical normal data. As illustrated in Fig. 1, atypical normal samples are forced to distribute far away from typical normal data. Hence, we assume that the unknown abnormal data that behave more like these atypical normal data will also lie far from the normal data.

II-C3 Reconstruction information

While transforming raw data to the latent space, the information contained in high-dimensional data should be retained in order to ensure that the latent space is a compressed representation of the raw data. Thus, we utilize the autoencoder structure as constraints on maintaining high-dimensional information. Equivalent to the ordinary autoencoder, the reconstruction ability of latent representations is quantified by a reconstruction loss.

II-D Training

As closeness and dispersion are opposite to each other, we designed a joint training procedure for feature learning. It is performed by three stages using the loss functions defined in Section II-E.

II-D1 First Stage

All data from the training set is used to train an autoencoder only with a regular reconstruction loss $\mathcal{L}_{\mathrm{rec}}$. During this stage, the network is considered to be a deep convolutional autoencoder. Note that no constraints are performed on the latent representations at this stage.

II-D2 Second Stage

Once the network is trained at the first stage, the similarity between the reconstructed data and the original data is calculated. Afterwards, according to a predefined ratio $\rho$, the first $\rho\%$ of the data with the lowest similarity scores are chosen as the atypical normal samples, whereas the remaining data is typical normal.

II-D3 Third Stage

To learn proper latent representations, the model is trained with different objectives regarding typical and atypical normal samples, respectively.

More precisely, the closeness loss $\mathcal{L}_{\mathrm{cls}}$ acts as the objective for only typical normal samples to force the corresponding latent representations to be close to each other. In contrast, one dispersion loss $\mathcal{L}_{\mathrm{disp},1}$ is designed for only atypical normal samples to keep them far away from each other in the latent space. Another dispersion loss $\mathcal{L}_{\mathrm{disp},2}$ is taken as an objective by both typical and atypical normal samples to maximize the distances between typical and atypical latent representations. Moreover, a regular reconstruction loss is applied as the objective for all the training samples to retain the high-dimensional information.

The two different training subsets, typical and atypical normal samples, have their own objectives. Nevertheless, all loss terms are reformulated in one unified loss function defined as

[TABLE]

where $\alpha$, $\beta_{1}$ and $\beta_{2}$ balance the ratio between all four loss terms.

II-E Loss Functions

First, the notations used in the loss functions are introduced.

• •

The tensor $\bm{X}$ denotes a batch of images from the training set. $B$ is the number of images in one minibatch. $\bm{X}_{j}$ represents the $j$-th image in the given minibatch. $\bm{x}_{j}$ is the vectorized form of $\bm{X}_{j}$.

• •

The matrix $\bm{Z}$ denotes a batch of latent representations with $L$ being the dimension of the latent vectors. The vector $\bm{z}_{j}$ is the $j$-th latent representation of $\bm{Z}$.

• •

The mapping from the raw data space to the latent space performed by the encoder is denoted as $f_{\mathrm{enc}}(\cdot)$. Accordingly, the decoder’s mapping from the latent space to the original data space is denoted as $f_{\mathrm{dec}}(\cdot)$. The function $f(\cdot)$ describes a cascade of these two mappings.

Based on these notations, we define three loss functions for each of the three characteristics of the latent space described in Section II-C.

II-E1 Reconstruction Loss

An ordinary mean squared error (MSE) loss is used as reconstruction loss

[TABLE]

where $\hat{\bm{x}}_{j}=f_{\mathrm{dec}}(f_{\mathrm{enc}}(\bm{x}_{j}))$ is the reconstruction of $\bm{x}_{j}$. The minimization of $\mathcal{L}_{\mathrm{rec}}$ forces the original data to be well reconstructed from the latent space. Correspondingly, the high-dimensional information is retained during the training. Moreover, this term helps to avoid trivial solutions, e.g. all-zero latent representations.

II-E2 Closeness Loss

The closeness requirement states that each typical normal latent representation $\bm{z}_{j}$ should have a small distance to any another randomly chosen typical normal latent representation $\bm{z}_{i\neq j}$, where arbitrary distance metrics can be used for this purpose. Specifically, it is a metric based on the Euclidean distance

[TABLE]

with $\bm{z}_{j}=f_{\mathrm{enc}}(\bm{X}_{j})$ and $\bm{z}_{i\neq j}=f_{\mathrm{enc}}(\bm{X}_{i\neq j})$. This minimization has the same expectation as if we calculated all distances from other latent representations, since the mini-batch stochastic gradient decent method is used in this work.

The above loss function alone has the tendency to map all raw data, both from the normal class and the abnormal classes, to a small region in the latent space. Fig. 3 shows this effect for the dataset Fashion-MNIST. As a result, latent representations of normal and abnormal data distribute in a mixture.

II-E3 Dispersion Loss

The dispersion loss between these typical and atypical normal subsets is defined as

[TABLE]

where $\bm{z}_{j,\mathrm{atypical}}=f_{\mathrm{enc}}(\bm{X}_{j,\mathrm{atypical}})$ and $\bm{z}_{j,\mathrm{typical}}=f_{\mathrm{enc}}(\bm{X}_{j,\mathrm{typical}})$. $\bm{X}_{j,\mathrm{typical}}$ denotes randomly chosen typical normal samples. The minimization of $\mathcal{L}_{\mathrm{disp},1}$ forces latent representations of atypical normal data to be far away from each other which results in a dispersion among atypical normal latent representations. Moreover, minimizing $\mathcal{L}_{\mathrm{disp},2}$ leads to large distances between typical and atypical normal samples.

II-F Network Architecture

The proposed feature learning method is modeled by a deep convolutional autoencoder based on AlexNet [9]. Both encoder and decoder are used during the training procedure. Once the model is trained, only the encoder part is utilized as a feature extractor.

III Experiments

The proposed method was used to extract latent features of different datasets which were subsequently fed into a one-class classifier. OCSVMs achieved state-of-the-art performance on extracted features, so an OCSVM was used to perform one-class classification. In one experiment, images of one class were selected as normal data, while the images of all remaining classes were considered as abnormal data and were not available during training. Each experiment was repeated five times with different initializations.

III-A Experimental Setup

III-A1 Datasets

The proposed method was evaluated on the datasets MNIST [10], Fashion-MNIST (FMNIST) [11] and CIFAR-10 [12]. All three datasets are composed of 10 different classes. The number of training data for all experiments was set to 4000. The numbers of normal and abnormal samples in the test set were 1000 and 9000, respectively111The MNIST dataset has about but not exactly 1000 normal and 9000 abnormal samples for each individual experiment.. For example, in one individual experiment, if digit 2 from dataset MNIST was considered as normal data, then the training set consisted of 4000 different images of digit 2. Furthermore, the test set consisted of 9000 other images of digit 2 and 1000 images of the other nine digits from 0 to 9 except of digit 2.222There are 20% samples of the test set randomly selected as validation set to choose a proper classification threshold for the metric balanced accuracy.

Before training, all image pixels were normalized to the range $[0,1]$ by min-max scaling. Note that images from MNIST and Fashion-MNIST have the size of $28\times 28$, while images from CIFAR-10 have the size of $32\times 32\times 3$.

III-A2 Hyperparameters

The proposed model was implemented with TensorFlow and Keras [13]. L2-regularization was used for every convolutional layer with a regularization parameter of $10^{-6}$. The training minibatch size was 64 for all experiments. The ratio $\rho$ for choosing atypical normal samples was set to 10, unless otherwise stated. $\alpha$, $\beta_{1}$ and $\beta_{2}$ defined in Section II-E were chosen to be 1, $10^{-5}$ and $10^{-5}$, unless otherwise specified. The dimension of the latent space was set to 64. The one-class classifier used in this work was the one-class support vector machine (OCSVM) with $\nu=0.1$ and $\gamma=\frac{1}{\#\mathrm{features}}$.

III-A3 Baseline Models

To the best of our knowledge, few prior works were designed for image-level one-class feature extraction. Therefore, the following conventional feature extraction methods were used as shallow baseline models:

• •

Original features (Original): The original images were vectorized as the input for OCSVM. Correspondingly, samples from MNIST and FMNIST have 784 features and those from CIFAR-10 have 3072 features.

• •

Principle Component Analysis (PCA): The first 64 PCA components of each input image were used as the input for OCSVM.

• •

Histogram of oriented gradients (HOG): HOG features [14] for each sample were extracted as the input for OCSVM. The length of HOG features for the samples from MNIST and FMNIST was 144 and for CIFAR-10 was 324.

Furthermore, the following deep feature extraction methods were considered as baselines in this work:

• •

Pretrained Features (ImageNet): Features extracted by a VGG19 [15] pretrained on ImageNet [16] were used as the input for OCSVM.

• •

Convolutional Autoencoder (CAE): Features extracted by a regular autoencoder without any constraints on the latent space were used for subsequent one-class classification.

• •

Autoencoder with closeness loss (CLS): An autoencoder was trained with the proposed closeness loss as a regularization but without intra-class splitting to extract features for one-class classification.

Note that both the baselines CAE and CLS shared the same architecture with the proposed method.

III-A4 Metrics

In one-class classification, imbalanced data is common. Hence, we used balanced accuracy as metric, because it allows a fair comparison between our method and the baseline models also for imbalanced datasets. The balanced accuracy is defined as

[TABLE]

with the terms true positive (TP), true negative (TN), false negative (FN) and false positive (FP). In this work, the normal class (the known class during training) was considered as the negative class. In contrast, the abnormal classes (unknown classes during training) were considered as positive classes.

III-B Results and Discussion

Table I shows the results averaged over all classes. Per class, the performance was determined by the mean and standard deviation of the balanced accuracies for different initializations. The results indicate that the proposed method performed best compared to all baseline models on all datasets. In particular, our method significantly outperformed the deep baseline models, i.e. the ImageNet and CAE features. Although CLS, a regular autoencoder only with the proposed closeness regularization, achieved a good performance, adding an additional dispersion constraint further improved the performance in all experiments.

Fig. 4 shows the balanced accuracies averaged over classes in relation to different ratios $\rho$. In general, the proposed method was not sensitive to the choice of the ratio $\rho$ for splitting the training data. However, the balanced accuracies tended to be lower if the ratio was higher, e.g. $\rho=50$, because a larger ratio indicates that more samples are considered to be atypical normal. This will lead to a low true negative rate.

Fig. 5 illustrates the balanced accuracies averaged over classes in relation to different values of $\beta_{1}$ and $\beta_{2}$. In the range of $10^{-7}$ to $10^{-3}$, the performance of the proposed method was stable for the conducted experiments. In contrast, the balanced accuracies on MNIST and FMNIST decreased with $\beta_{1}=\beta_{2}=10^{-1}$, because the dispersion loss term is dominant. This is not desirable for an OCSVM.

IV Conclusion

In this work, we presented a novel generic feature learning method for one-class classification. To learn a proper latent space, normal training samples were split into typical and atypical normal data. These two subsets were used to train a network with different losses in an joint way under the constraints of the proposed closeness and dispersion requirements. As a result, the trained feature extractor enabled to extract highly discriminative features between normal and abnormal data. Various and intensive experiments on three image datasets used the extracted features as the input for an OCSVM to perform one-class classification. In all conducted experiments, the method outperformed all other baseline models. In particular, our method showed a large improvement over ImageNet features, CAE and the original features. An ongoing work is to realize the proposed method in an end-to-end way since it has already shown its efficient feature extraction ability for one-class classification.