C2AE: Class Conditioned Auto-Encoder for Open-set Recognition

TL;DR

This paper introduces C2AE, a class conditioned auto-encoder approach for open-set recognition, effectively distinguishing known from unknown classes by combining classification and reconstruction error modeling.

Contribution

The paper proposes a novel training and testing methodology for open-set recognition using class conditioned auto-encoders and EVT-based thresholding, outperforming existing methods.

Findings

Significantly better performance than state-of-the-art methods.

Effective identification of unknown classes in image datasets.

Robust open-set recognition across multiple datasets.

Abstract

Models trained for classification often assume that all testing classes are known while training. As a result, when presented with an unknown class during testing, such closed-set assumption forces the model to classify it as one of the known classes. However, in a real world scenario, classification models are likely to encounter such examples. Hence, identifying those examples as unknown becomes critical to model performance. A potential solution to overcome this problem lies in a class of learning problems known as open-set recognition. It refers to the problem of identifying the unknown classes during testing, while maintaining performance on the known classes. In this paper, we propose an open-set recognition algorithm using class conditioned auto-encoders with novel training and testing methodology. In contrast to previous methods, training procedure is divided in two sub-tasks, 1. closed-set classification and, 2. open-set identification (i.e. identifying a class as known or unknown). Encoder learns the first task following the closed-set classification training pipeline, whereas decoder learns the second task by reconstructing conditioned on class identity. Furthermore, we model reconstruction errors using the Extreme Value Theory of statistical modeling to find the threshold for identifying known/unknown class samples. Experiments performed on multiple image classification datasets show proposed method performs significantly better than state of the art.

Figures

Tables

Table 1. Table 1 : AUROC for open-set identification, values other than the proposed method are taken from [ 24 ] .

Method	MNIST	SVHN	CIFAR10	CIFAR+10	CIFAR+50	TinyImageNet
SoftMax	0.978	0.886	0.677	0.816	0.805	0.577
OpenMax [4] (CVPR’16)	0.981	0.894	0.695	0.817	0.796	0.576
G-OpenMax [8] (BMVC’17)	0.984	0.896	0.675	0.827	0.819	0.580
OSRCI [24] (ECCV’18)	0.988	0.910	0.699	0.838	0.827	0.586
Proposed Method	0.989	0.922	0.895	0.955	0.937	0.748

Table 2. Table 2 : AUROC for open-set identification, values other than the proposed method are taken from [ 24 ] . Standard deviation value for state of the art not available for CIFAR+10, CIFAR+50 and TinyImageNet.

Method	MNIST	SVHN	CIFAR10	CIFAR+10	CIFAR+50	TinyImageNet
SoftMax	0.978 $\pm$ 0.002	0.886 $\pm$ 0.006	0.677 $\pm$ 0.032	0.816 $\pm$ –	0.805 $\pm$ –	0.577 $\pm$ –
OpenMax (CVPR’16)	0.981 $\pm$ 0.002	0.894 $\pm$ 0.008	0.695 $\pm$ 0.032	0.817 $\pm$ –	0.796 $\pm$ –	0.576 $\pm$ –
G-OpenMax (BMVC’17)	0.984 $\pm$ 0.001	0.896 $\pm$ 0.006	0.675 $\pm$ 0.035	0.827 $\pm$ –	0.819 $\pm$ –	0.580 $\pm$ –
OSRCI (ECCV’18)	0.988 $\pm$ 0.001	0.910 $\pm$ 0.006	0.699 $\pm$ 0.029	0.838 $\pm$ –	0.827 $\pm$ –	0.586 $\pm$ –
Proposed Method	0.989 $\pm$ 0.002	0.922 $\pm$ 0.009	0.895 $\pm$ 0.008	0.955 $\pm$ 0.006	0.937 $\pm$ 0.004	0.748 $\pm$ 0.005

Equations

$${\mathcal{L}}_c(\{{\Theta }_f,{\Theta }_c\})=-\frac{1}{N}\sum _{i=1}^N\sum _{j=1}^k{\mathbb{I}}_{y_i}(j)\text{log}[p_{y_i}(j)],$$

$${\gamma }_j$$

$$z_{l_j}$$

$$l_j(x)=\{\begin{array}{l}+1,x=j,\\ -1,x\ne j,\end{array}x,j\in \{1,2,...,k\}.$$

$$z_i$$

$${\gamma }_{y_i^m}$$

$${\beta }_{y_i^m}$$

$$z_{i{l}_m}$$

$${\tilde{X}}_i^m$$

$${\mathcal{L}}_r^m(\{{\Theta }_g,{\Theta }_{\gamma },{\Theta }_{\beta }\})=\frac{1}{N}\sum _{i=1}^N||X_i-{\tilde{X}}_i^m|{|}_1,$$

$${\mathcal{L}}_r^{nm}(\{{\Theta }_g,{\Theta }_{\gamma },{\Theta }_{\beta }\})=\frac{1}{N}\sum _{i=1}^N||X_i^{nm}-{\tilde{X}}_i^{nm}|{|}_1,$$

$$\underset{\{{\Theta }_g,{\Theta }_{\gamma },{\Theta }_{\beta }\}}{\min }\alpha {\mathcal{L}}_r^m(\{{\Theta }_g,{\Theta }_{\gamma },{\Theta }_{\beta }\})$$

$$+(1-\alpha )$$

$$F_U(w)=\mathcal{P}(w-u\le w|w>u)=\frac{F_W(u+w)-F_W(u)}{1-F_W(u)},$$

$$G(w;\zeta ,\mu )=\{\begin{array}{l}1-(1+\frac{\zeta \cdot w}{\mu }{)}^{\frac{1}{\zeta }},\text{if}\zeta \ne 0,\\ 1-e^{\frac{w}{\mu }},\text{if}\zeta =0,\end{array}$$

$${\tilde{X}}_i^m$$

$${\tilde{X}}_i^{nm}$$

$$S_m$$

$$S_{nm}$$

$$\forall i$$

$${\tau }^{\ast }=\underset{\tau }{\min }$$

$$=\underset{\tau }{\min }$$

$$\mathbf{O}=1-\sqrt{\frac{2\times N_{train}}{N_{test}+N_{target}}},$$

Full text

1 Introduction

Recent advancements in computer vision have resulted in significant improvements for image classification systems [13], [19], [14], [39]. Especially the rise of Deep Convolutional Neural Network has resulted in classification error rates surpassing the human-level performance [12]. These promising results, enable their potential use in many real world applications. However, when deployed in a real world scenario, such systems are likely to observe samples from classes not seen during training (i.e. unknown classes also referred as “unknown unknowns” [35]). Since, the traditional training methods follow this closed-set assumption, the classification systems observing any unknown class samples are forced to recognize it as one of the known classes. As a result, it affects the performance of these systems, as evidenced by Jain et al. with digit recognition example. Hence, it becomes critical to correctly identify test samples as either known or unknown for a classification model. This problem setting of identifying test samples as known/unknown and simultaneously correctly classifying all of known classes, is referred to as open-set recognition [35]. Fig. 1 illustrates a typical example of classification in the open-set problem setting.

In an open-set problem setting, it becomes challenging to identify unknown samples due to the incomplete knowledge of the world during training (i.e. only the known classes are accessible). To overcome this problem many open-set methods in the literature [4], [36], [40], [38] adopt recognition score based thresholding models. However, when using these models one needs to deal with two key questions, * 1) what is a good score for open-set identification?* (i.e., identifying a class as known or unknown), and given a score, * 2) what is a good operating threshold for the model?*. There have been many methods that explore these questions in the context of traditional methods such as Support Vector Machines [35], [36], Nearest Neighbors [16], [3] and Sparse Representation [40]. However, these questions are relatively unexplored in the context of deep neural networks [38], [4], [24], [8], [7].

Even-though deep neural networks are powerful in learning highly discriminative representations, they still suffer from performance degradation in the open-set setting [4]. In a naive approach, one could apply a thresholding model on SoftMax scores. However, as shown by experiments in [4], that model is sub-optimal for open-set identification. A few methods have been proposed to better adapt the SoftMax scores for open-set setting. Bendale et al. proposed a calibration strategy to update SoftMax scores using extreme value modeling [4]. Other strategies, Ge et al. [8] and Lawrence et al. [24] follow data augmentation technique using Generative Adversarial Networks (GANs) [10]. GANs are used to synthesize open-set samples and later used to fine-tuning to adapt SoftMax/OpenMax scores for open-set setting. Shu et al. [38] introduced a novel sigmoid-based loss function for training the neural network to get better scores for open-set identification.

All of these methods modify the SoftMax scores, so that it can perform both open-set identification and maintain its classification accuracy. However, it is extremely challenging to find a single such score measure, that can perform both. In Contrast to these methods, in proposed approach the training procedure for open-set recognition using class conditional auto-encoders, is divided it into two sub-tasks, 1. closed-set classification, and 2. open-set identification. These sub-tasks are trained separately in a stage-wise manner. Experiments show that such approach provides good open-set identification scores and it is possible to find a good operating threshold using the proposed training and testing strategy.

In summary, this paper makes following contributions,

• •

A novel method for open-set recognition is proposed with novel training and testing algorithm based on class conditioned auto-encoders.

• •

We show that dividing open-set problem in sub-tasks can help learn better open-set identification scores.

• •

Extensive experiments are conducted on various image classification datasets and comparisons are performed against several recent state-of-the-art approaches. Furthermore, we analyze the effectiveness of the proposed method through ablation experiments.

2 Related Work

Open-set Recognition. The open-set recognition methods can be broadly classified in to two categories, traditional methods and neural network-based methods. Traditional methods are based on classification models such as Support Vector Machines (SVMs), Nearest Neighbors, Sparse Representation etc. Scheirer et al. [36] extended the SVM for open-set recognition by calibrating the decision scores using the extreme value distribution. Specifically, Scheirer et al. [36] utilized two SVM models, one for identifying a sample as unknown (referred as CAP models) and other for traditional closed-set classification. PRM Junior et al. [15] proposed a nearest neighbor-based open-set recognition model utilizing the neighbor similarity as a score for open-set identification. PRM Junior et al. later also presented specialized SVM by constraining the bias term to be negative. This strategy in the case of Radial Basis Function kernel, yields an open-set recognition model. Zhang et al. [40] proposed an extension of the Sparse Representation-based Classification (SRC) algorithm for open-set recognition. Specifically, they model residuals from SRC using the Generalized-Pareto extreme value distribution to get score for open-set identification.

In neural network-based methods, one of the earliest works by Bendale et al. [4] introduced an open-set recognition model based on “activation vectors” (i.e. penultimate layer of the network). Bendale et al. utilized meta-recognition for multi-class classification by modeling the distance from “mean activation vector” using the extreme value distribution. SoftMax scores are calibrated using these models for each class. These updated scores, termed as OpenMax, are then used for open-set identification. Ge et al. [8] introduced a data augmentation approach called G-OpenMax. They generate unknown samples from the known class training data using GANs and use it to fine-tune the closed-set classification model. This helps in improving the performance for both SoftMax and OpenMax based deep network. Along the similar motivation, Neal et al. [24] proposed a data augmentation strategy called counterfacutal image generation. This strategy also utilizes GANs to generate images that resemble known class images but belong to unknown classes. In another approach, Shu et al. [38] proposed a $k$-sigmoid activation-based novel loss function to train the neural network. Additionally, they perform score analysis on the final layer activations to find an operating threshold, which is helpful for open-set identification. There are some variation of open-set recognition by relaxing its formulation in the form of anomaly detection [26], [27], [30] and novelty detection [31], [28] etc, but for this paper we only focus on the general open-set recognition problem.

Extreme Value Theory. Extreme value modeling is a branch of statistics that deals with modeling of statistical extremes. The use of extreme value theory in vision tasks largely deals with post recognition score analysis [29], [36]. Often for a given recognition model the threshold to reject/accept lies in the overlap region of extremes of match and non-match score distributions [37]. In such cases, it makes sense to model the tail of the match and non-match recognition scores as one of the extreme value distributions. Hence, many visual recognition methods including some described above, utilize extreme value models to improve the performance further [40], [36]. In the proposed approach as well, the tail of open-set identification scores are modeled using the extreme value distribution to find the optimal threshold for operation.

3 Proposed Method

The proposed approach divides the open-set recognition problem into two sub-tasks, namely, closed-set classification and open-set identification. The training procedure for these tasks are shown in Fig. 2 as stage-1 and stage-2. Stage-3 in Fig. 2 provides overview of the proposed approach at inference. In what follows, we present details of these stages.

3.1 Closed-set Training (Stage 1)

Given images in a batch $\{X_{1},X_{2},...,X_{N}\}\in\mathcal{K}$, and their corresponding labels $\{y_{1},y_{2},...,y_{N}\}$. Here $N$ is the batch size and $\forall y_{i}\in\{1,2,..,k\}$. The encoder ($\mathcal{F}$) and the classifier ($\mathcal{C}$) with parameters $\Theta_{f}$ and $\Theta_{c}$, respectively are trained using the following cross entropy loss,

[TABLE]

where, $\mathbb{I}_{y_{i}}$ is an indicator function for label $y_{i}$ (i.e., one hot encoded vector) and $p_{y_{i}}=\mathcal{C}(\mathcal{F}(X_{i}))$ is a predicted probability score vector. $p_{y_{i}}(j)$ is probability of the $i^{th}$ sample being from the $j^{th}$ class.

3.2 Open-set Training (Stage 2)

There are two major parts in open-set training, conditional decoder training, followed by EVT modeling of the reconstruction errors. In this stage, the encoder and classifier weights are fixed and don’t change during optimization.

3.2.1 Conditional Decoder Training

For any batch described in Sec. 3.1, $\mathcal{F}$ is used to extract the latent vectors as, $\{z_{1},z_{2},...,z_{N}\}$. This latent vector batch is conditioned following the work by Perez et al. [32] called FiLM. FiLM influences the input feature map by applying a feature-wise linear modulations (hence the name FiLM) based on conditioning information. For a input feature $z$ and vector $l_{j}$ containing conditioning information can be given as,

[TABLE]

where,

[TABLE]

Here, $H_{\gamma}$ and $H_{\beta}$ are neural networks with parameters $\Theta_{\gamma}$ and $\Theta_{\beta}$. Tensors $z_{l_{j}}$, $\gamma_{j}$, $\beta_{j}$ have the same shape and $\odot$ represents the Hadamard product. $l_{j}$ is used for conditioning, and referred to as label condition vector in the paper. Also, the notation $z_{l_{j}}$ is used to describe the latent vector $z$ conditioned on the label condition vector $l_{j}$, i.e, $z|{l_{j}}$.

The decoder ($\mathcal{G}$ with parameters $\Theta_{g}$) is expected to perfectly reconstruct the original input when conditioned on the label condition vector matching the class identity of the input, referred here as the match condition vector ($l_{m}$), can be viewed as a traditional auto-encoder. However, here $\mathcal{G}$ is additionally trained to poorly reconstruct the original input when conditioned on the label condition vector, that does not match the class identity of the input, referred here as the non-match condition vector ($l_{nm}$). The importance of this additional constraint on the decoder is discussed in Sec. 3.2.2 while modeling the reconstruction errors using EVT. For the rest of this paper, we use superscript $m$ and $nm$ to indicate match and non-match, respectively.

Now, for a given input $X_{i}$ from the batch and $l_{m}=l_{y^{m}_{i}}$ and $l_{nm}=l_{y_{j}^{nm}}$, for any random $y^{nm}_{i}\neq y_{i}$ sampled from $\{1,2,..,k\}$, be its corresponding match and non-match condition vectors, the feed forward path for stage-2 can be summarized through the following equations,

[TABLE]

Following the above feed-forward path, the loss functions in the second stage of training to train the decoder ($\mathcal{G}$ with parameters $\Theta_{g}$) and conditioning layer (with parameters $\Theta_{\gamma}$ and $\Theta_{\beta}$) are given as follows,

[TABLE]

[TABLE]

[TABLE]

Here, the loss function $\mathcal{L}^{m}_{r}$ corresponds to the constraint that output generated using match condition vector $\tilde{X}^{m}_{i}$, should be perfect reconstruction of $X_{i}$. Whereas, the loss function $\mathcal{L}^{nm}_{r}$ corresponds to the constraint that output generated using non match condition vector $\tilde{X}^{nm}_{i}$, should have poor reconstruction. To enforce the later condition, another batch $\{X_{1}^{nm},X_{2}^{nm},...,X_{N}^{nm}\}$, is sampled from the training data, such that new batch does not have class identity consistent with the match condition vector. This in effect achieves the goal of poor reconstruction when conditioned $l_{y^{nm}_{i}}$. This conditioning strategy in a way, emulates open-set behavior (as will be discussed further in Sec. 3.2.2). Here, the network is specifically trained to produce poor reconstructions when class identity of an input image does not match the condition vector. So, when encountered with an unknown class test sample, ideally none of the condition vector would match the input image class identity. This will result in poor reconstruction for all condition vectors. While, when encountered with the known test sample, as one of the condition vector will match the input image class identity, it will produce a perfect reconstruction for that particular condition vector. Hence, training with the non-match loss helps the network adapt better to open-set setting. Here, $\mathcal{L}^{nm}_{r}$ and $\mathcal{L}^{m}_{r}$ are weighted with $\alpha\in[0,1]$.

3.2.2 EVT Modeling

Extreme Value Theory. Extreme value theory is often used in many visual recognition systems and is an effective tool for modeling post-training scores [36], [37]. It has been used in many applications such as finance, railway track inspection etc. [23], [1], [9] as well as open-set recognition [4], [36], [40]. In this paper we follow the Picklands-Balkema-deHaan formulation [33], [2] of the extreme value theorem. It considers modeling probabilities conditioned on random variable exceeding a high threshold. For a given random variable $W$ with a cumulative distribution function (CDF) $F_{W}(w)$ the conditional CDF for any $w$ exceeding the threshold $u$ is defined as,

[TABLE]

where, $\mathcal{P}(\cdot)$ denotes probability measure function. Now, given I.I.D. samples, $\{W_{i},...,W_{n}\}$, the extreme value theorem [33] states that, for large class of underlying distributions and given a large enough $u$, $F_{U}$ can be well approximated by the Generalized Pareto Distribution (GPD),

[TABLE]

such that $-\infty<\zeta<+\infty$, $0<\mu<+\infty$, $w>0$ and $\zeta w>-\mu$. $G(.)$ is CDF of GPD and for $\zeta=0$, reduces to the exponential distribution with parameter $\mu$ and for $\zeta\neq 0$ takes the form of Pareto distribution [6].

Parameter Estimation. When modeling the tail of any distribution as GPD, the main challenge is in finding the tail parameter $u$ to get the conditional CDF. However, it is possible to find an estimated value of $u$ using mean excess function (MEF), i.e., $E[W-u|W>u]$ [37]. It has been shown that for GPD, MEF holds a linear relationship with $u$. Many researchers use this property of GPD to estimate the value of $u$ [37], [29]. Here, the algorithm for finding $u$, introduced in [29] for GPD is adopted with minor modifications. See [29], [37] for more details regarding MEF or tail parameter estimation. After getting an estimate for $u$, since from extreme value theorem [33], we know that set $\{w\in W\ |\ w>u\}$, follows GPD distribution, rest of the parameters for GPD, i.e. $\zeta$ and $\mu$ can be easily estimated using the maximum likelihood estimation techniques [11], except for some rarely observed cases [5].

3.2.3 Threshold Calculation

After training procedure described in previous sections, Sec. 3.1 and Sec. 3.2, set of match and non-match reconstruction errors are created from training set, $\{X_{1},X_{2},...,X_{N_{train}}\}\in\mathcal{K}$, and their corresponding match and non match labels, $\{y^{m}_{1},y^{m}_{2},...,y^{m}_{N_{train}}\}$ and $\{y_{1}^{nm},y_{2}^{nm},...,y_{N_{train}}^{nm}\}$. Let, $r^{m}_{i}$ be the match reconstruction error and $r^{nm}_{i}$ be the non match reconstruction error for the input $X_{i}$, then the set of match and non match errors can be calculated as,

[TABLE]

[TABLE]

Typical histograms of $S_{m}$ (set of match reconstruction errors) and $S_{nm}$ (set of non-match reconstruction errors) are shown in Fig. 3(a). Note that the elements in these sets are calculated solely based on what is observed during training (i.e., without utilizing any unknown samples). Fig. 3(b) shows the normalized histogram of the reconstruction errors observed during inference from the test samples of known class set $(\mathcal{K})$, and unknown class set $(\mathcal{U})$. Comparing these figures in Fig. 3, it can be observed that the distribution of $S_{m}$ and $S_{nm}$ computed during training, provides a good approximation for the error distributions observed during inference, for test samples from known set $(\mathcal{K})$ and unknown set $(\mathcal{U})$. This observation also validates that non match training emulates an open-set test scenario (also discussed in Sec. 3.2) where the input does not match any of the class labels. This motivates us to use $S_{m}$ and $S_{nm}$ to find an operating threshold for open-set recognition to make a decision about any test sample being known/unknown.

Now, It is safe to assume that the optimal operating threshold ($\tau^{*}$) lies in the region $S_{m}\cap S_{nm}$. Here, the underlying distributions of $S_{m}$ and $S_{nm}$ are not known. However, as explained in 3.2.2, it is possible to model the tails of $S_{m}$ (right tail) and $S_{nm}$ (left tail) with GPD as $G_{m}$ and $G_{nm}$ with $G(\cdot)$ being a CDF. Though, GPD is only defined for modeling maxima, before fitting $G_{nm}$ left tail of $S_{nm}$ we perform inverse transform as $S_{nm}^{\prime}=-S_{nm}$. Assuming the prior probability of observing unknown samples is $p_{u}$, the probability of errors can be formulated as a function of the threshold $\tau$,

[TABLE]

Solving the above equation should give us an operating threshold that can minimize the probability of errors for a given model and can be solved by a simple line search algorithm by searching for $\tau^{*}$ in the range $\{S_{m}\cap S_{nm}\}$. Here, the accurate estimation of $\tau^{*}$ depends on how well $S_{m}$ and $S_{nm}$ represent the known and unknown error distributions. It also depends on the prior probability $p_{u}$, effect of this prior will be further discussed in Sec. 4.3.

3.3 Open-set Testing by k-inference (Stage 3)

Here, we introduce the open-set testing algorithm for proposed method. The testing procedure is described in Algo. 1 and an overview of this is also shown in Fig. 2. This testing strategy involves conditioning the decoder $k$-times with all possible condition vectors to get $k$ reconstruction errors. Hence, it is referred as $k$-inference algorithm.

4 Experiments and Results

In this section we evaluate the performance of the proposed approach and compare it with the state of the art open-set recognition methods. The experiments in Sec. 4.2, we measure the ability of algorithm to identify test samples as known or unknown without considering operating threshold. In second set of experiments in Sec. 4.3, we measure overall performance (evaluated using F-measure) of open-set recognition algorithm. Additionally through ablation experiments, we analyze contribution from each component of the proposed method.

4.1 Implementation Details

We use Adam optimizer [17] with learning rate $0.0003$ and batch size, $N$=$64$. The parameter $\alpha$, described in Sec. 3.2, is set equal to 0.9. For all the experiments, conditioning layer networks $\mathcal{H}_{\gamma}$ and $\mathcal{H}_{\beta}$ are a single layer fully connected neural networks. Another important factor affecting open-set performance is openness of the problem. Defined by Scheirer et al. [35], it quantifies how open the problem setting is,

[TABLE]

where, $N_{train}$ is the number of training classes seen during training, $N_{test}$ is the number of test classes that will be observed during testing, $N_{target}$ is the number of target classes that needs to be correctly recognized during testing. We evaluate performance over multiple openness value depending on the experiment and dataset.

4.2 Experiment I : Open-set Identification

The evaluation protocol defined in [24] is considered and area under ROC (AUROC) is used as evaluation metric. AUROC provides a calibration free measure and characterizes the performance for a given score by varying threshold. The encoder, decoder and classifier architecture for this experiment is similar to the architecture used by [24] in their experiments. Following the protocol in [24], we report the AUROC averaged over five randomized trials.

4.2.1 Datasets

Here, we provide summary of these protocols for each dataset,

MNIST, SVHN, CIFAR10. For MNIST [21], SVHN [25] and CIFAR10 [18], openness of the problem is set to $\mathbf{O}=13.39\%$, by randomly sampling 6 known classes and 4 unknown classes.

CIFAR+10, CIFAR+50. For CIFAR+$M$ experiments, 4 classes are sampled from CIFAR10 for training. $M$ non overlapping classes are used as the unknowns, which are sampled from the CIFAR100 dataset [18]. Openness of the problem for CIFAR+10 and CIFAR+50 is $\mathbf{O}=33.33\%$ and $62.86\%$, respectively.

TinyImageNet. For experiments with the TinyImageNet [20], 20 known classes and 180 unknown classes with openness $\mathbf{O}=57.35\%$ are randomly sampled for evaluation.

4.2.2 Comparison with state-of-the-art

For comparing the open-set identification performance, we consider the following methods:

I. SoftMax : SoftMax score of a predicted class is used for open-set identification.

II. OpenMax [4]: The score of $k$+$1^{th}$ class and score of the predicted class is used for open-set identification.

III. G-OpenMax [8]: It is a data augmentation technique, which utilizes the OpenMax scores after training the network with the generated data.

IV. OSRCI [24]: Another data augmentation technique called counterfactual image generation is used for training the network for $k$+$1$ class classification. We refer to this method as Open-set Recognition using Counterfactual Images (OSRCI). The score value $P(y_{k+1})-\underset{i\leq k}{max}\ P(y_{i})$ is used for open-set identification.

Results corresponding to this experiment are shown in Table 2. As seen from this table, the proposed method outperform the other methods, showing that open-set identification training in proposed approach learns better scores for identifying unknown classes. From the results, we see that our method on the digits dataset produces a minor improvement compared to the other recent methods. This is mainly do the reason that results on the digits dataset are almost saturated. On the other hand, our method performs significantly better than the other recent methods on the object datasets such as CIFAR and TinyImageNet.

4.3 Experiment II : Open-set Recognition

This experiment shows the overall open-set recognition performance evaluated with F-measure. For this experiment we consider LFW Face dataset [22]. We extend the protocol introduced in [35] for open-set face recognition on LFW. Total 12 classes containing more than 50 images are considered as known classes and divided into training and testing split by 80/20 ratio. Image size is kept to 64$\times$64. Since, LFW has 5717 number of classes, we vary the openness from $0\%$ to $93\%$ by taking 18 to 5705 unknown classes during testing. Since, many classes contain only one image, instead of random sampling, we sort them according to the number of images per class and add it sequentially to increase the openness. It is obvious that with the increase in openness, the probability of observing unknown will also increase. Hence, it is reasonable to assume that prior probability $p_{u}$ will be a function of openness. For this experiment we set $p_{u}=0.5*\mathbf{O}$.

4.3.1 Comparison with state-of-the-art

For comparing the open-set recognition performance, we consider the following methods:

I. W-SVM (PAMI’14) : W-SVM is used as formulated in [35], which trains Weibull calibrated SVM classifier for open set recognition.

II. SROR (PAMI’16) : SROR is used as formulated in [40]. It uses sparse representation-based framework for open-set recognition.

III. DOC (EMNLP’16) : It utilizes a novel sigmoid-based loss function for training a deep neural network [38].

To have a fair comparison with these methods, we use features extracted from the encoder ($\mathcal{F}$) to train W-SVM and SROR. For DOC, the encoder ($\mathcal{F}$) is trained with the loss function proposed in [38]. Experiments on LFW are performed using a U-Net [34] inspired encoder-decoder architecture. More details regarding network architecture is included in the supplementary material.

Results corresponding to this experiment is shown in Fig.  4(a). From this figure, we can see that the proposed approach remains relatively stable with the increase in openness, outperforming all other methods. One interesting trend noticed here is, that DOC initially performs better than the statistical methods such as W-SVM and SROR. However with openness more than 50% the performance suffers significantly. While the statistical methods though initially perform poor compared to DOC, but remain relatively stable and performs better than DOC as the openness is increased (especially over $\mathbf{O}>$50%).

4.3.2 Ablation Study

In this section, we present ablation analysis of the proposed approach on the LFW Face dataset. The contribution of individual components to the overall performance of the method is reported by creating multiple baselines of the proposed approach. Starting with the most simple baseline, i.e., thresholding SoftMax probabilities of a closed-set model, each component is added building up to the proposed approach. Detailed descriptions of these baselines are given as follows,

I. CLS : Encoder $(\mathcal{F})$ and the classifier $(\mathcal{C})$ are trained for $k$-class classification. Samples with probability score prediction less than 0.5 are classified as unknown.

II. CLS+DEC : In this baseline, only the networks $\mathcal{F}$, $\mathcal{C}$ and the decoder $(\mathcal{G})$ are trained as described in Sec. 3, except $\mathcal{G}$ is only trained with match loss function, $\mathcal{L}_{r}^{m}$. Samples with more than 95% of maximum train reconstruction error observed, are classified as unknown.

III. Naive : Here, the networks $\mathcal{F}$, $\mathcal{C}$ and $\mathcal{G}$ and the conditioning layer networks ($\mathcal{H}_{\gamma}$ and $\mathcal{H}_{\beta}$) are trained as described in Sec. 3, but instead of modeling the scores using EVT as described in Sec. 3.2.2, threshold is directly estimated from the raw reconstruction errors.

IV. Proposed method (pu = 0.5) : $\mathcal{F}$, $\mathcal{C}$, $\mathcal{G}$ and condition layer networks ($\mathcal{H}_{\gamma}$ and $\mathcal{H}_{\beta}$) are trained as described in Sec. 3 and to find the threshold prior probability of observing unknown is set to $p_{u}=0.5$.

V. Proposed method: Method proposed in this paper, with $p_{u}$ set as described in Sec. 4.3.

Results corresponding to the ablation study are shown in Fig. 4(b). Being a simple SoftMax thresholding baseline, CLS has weakest performance. However, when added with a match loss function ($\mathcal{L}_{r}^{m}$) as in CLS+DEC, the open-set identification is performed using reconstruction scores. Since, it follows a heuristic way of thresholding, the performance degrades rapidly as openness increases. However, addition of non match loss function ($\mathcal{L}_{r}^{nm}$), as in the Naive baseline, helps find a threshold value without relying on heuristics. As seen from the Fig. 4(b) performance of Naive baseline remains relatively stable with increase in openness, showing the importance of loss function $\mathcal{L}_{r}^{nm}$. Proposed method with $p_{u}$ fixed to 0.5, introduces EVT modeling on reconstruction errors to calculate a better operating threshold. It can be seen from the Fig. 4(b), such strategy improves over finding threshold based on raw score values. This shows importance applying EVT models on reconstruction errors. Now, if $p_{u}$ is set to $0.5*\mathbf{O}$, as in the proposed method, there is a marginal improvement over the fixed $p_{u}$ baseline. This shows benefit of setting $p_{u}$ as a function of openness. It is interesting to note that for large openness values (as $0.5*\mathbf{O}\rightarrow 0.5$), both fixed $p_{u}$ baseline and proposed method achieve similar performance.

5 Conclusion

We presented an open-set recognition algorithm based on class conditioned auto-encoders. We introduced training and testing strategy for these networks. It was also shown that dividing the open-set recognition into sub tasks helps learn a better score for open-set identification. During training, enforcing conditional reconstruction constraints are enforced, which helps learning approximate known and unknown score distributions of reconstruction errors. Later, this was used to calculate an operating threshold for the model. Since inference for a single sample needs $k$ feed-forwards, it suffers from increased test time. However, the proposed approach performs well across multiple image classification datasets and providing significant improvements over many state of the art open-set algorithms. In our future research, generative models such as GAN/VAE/FLOW can be explored to modify this method. We will revise the manuscript with such details in the conclusion.

Acknowledgements

This research is based upon work supported by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via IARPA R&D Contract No. 2014-14071600012. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies or endorsements, either expressed or implied, of the ODNI, IARPA, or the U.S. Government.

6 Supplementary Material for C2AE: Class Conditioned Auto-Encoder for Open-set Recognition

This contains the supplementary material for the paper C2AE: Class Conditioned Auto-Encoder for Open-set Recognition. Due to the space limitations in the submitted paper, we provide some additional details regarding the proposed method.

6.1 Toy Examples

To see the decision boundaries learned using the proposed approach, we perform few experiments with 2-Dimensional toy data. For these experiments the encoder, decoder and classifier architectures are FC(2)-Sig-FC(5)-Sig, FC(5)-Sig-FC(2) and FC(5)-Sig-FC(2), respectively. Here, FC(T) indicates fully connected layer with T hidden units, Sig is the sigmoid activation. We train these networks using the proposed approach for three different variations of 2-Dimensional datasets, namely Two-Gauss, Four-Gauss and Uni-Gauss. Two-Gauss and Four-Gauss have two and four 2D Gaussians with different means and same variance, respectively. Whereas Uni-Gauss has one class as 2D Gaussian and another classes Uniformly distributed. As it can be seen from Fig. 5, the proposed approach is able to learn tight boundaries surrounding the data points and identify all of the remaining space as unknown.

6.2 Results

Here we present the AUROC table for open-set identification with standard deviation values. Standard deviation values were not available for CIFAR+10, CIFAR+50 and TinyImageNet as the values are taken from [24].

6.3 Histogram Progression

Fig. 6 and Fig. 7, provides evolution of reconstruction errors during the learning procedure. The reconstruction errors for match, non match, known and unknown are provided at iteration 1, 5k and 500k. As it can be seen from Fig. 6(a), since the network is initialized with random weights, the reconstruction errors for match and non match are not discriminative. However, since the network is trained to learn the discriminative reconstructions for match and non match conditioning, with the increase in iterations, the reconstruction errors become more discriminative as seen from the Fig. 6(b) and Fig. 6(c). As a result, known and unknown reconstruction errors follow the same trend as that of match and non match as evident from the Fig. 7. The SVHN dataset is used for generating the normalized histograms of match, non match, known and unknown data reconstruction errors.

6.4 Network Architecture

The network architecture for the LFW experiments is shown in the Fig. 8. It is an U-Net inspired network architecture with a FiLM conditioning layer in the middle. The network architecture is as follows,

C(64)-C(128)-C(256)-C(512)-C(1024)-FiLM-DC(2048)-DC(1024)-DC(512)-DC(256)-DC(124)-DC(3)-Tanh.

Here, C(T) represents T channel convolution layer followed by instance normalization and leaky ReLU activation. DC(T) represents T channel transposed convolution layer followed by instance normalization and Upsampling. FiLM layer is a conditioning layer which modulates feature maps from C(1024) with linear modulation parameters $\gamma_{c}$ and $\beta_{c}$ of size 1024$\times$2$\times$2, based on label conditioning vector. Here, the convolution blocks are used as encoder and deconvolution blocks are used as decoder. As explained in the proposed approach, the encoder weights are frozen during training in stage-2. The classifier network for the experiments with the LFW dataset is a single layer fully connected network with 12 hidden units (same as number of known classes).