Towards well-specified semi-supervised model-based classifiers via structural adaptation

TL;DR

This paper addresses the challenge of model misspecification in semi-supervised learning by proposing a structural adaptation method that detects and corrects generative model biases, improving classification accuracy.

Contribution

It introduces a criterion for detecting model misspecification and an automatic modification approach for generative models during training.

Findings

Outperforms state-of-the-art semi-supervised methods on PASCAL VOC'07.

Effectively detects model misspecification during training.

Improves classification accuracy through structural adaptation.

Abstract

Semi-supervised learning plays an important role in large-scale machine learning. Properly using additional unlabeled data (largely available nowadays) often can improve the machine learning accuracy. However, if the machine learning model is misspecified for the underlying true data distribution, the model performance could be seriously jeopardized. This issue is known as model misspecification. To address this issue, we focus on generative models and propose a criterion to detect the onset of model misspecification by measuring the performance difference between models obtained using supervised and semi-supervised learning. Then, we propose to automatically modify the generative models during model training to achieve an unbiased generative model. Rigorous experiments were carried out to evaluate the proposed method using two image classification data sets PASCAL VOC'07 and MIR Flickr. Our proposed method has been demonstrated to outperform a number of state-of-the-art semi-supervised learning approaches for the classification task.

Tables

Table 1. Table 1: Results on PASCAL Dataset

PASCAL07	without tag					with tag
	S4VM	co-training	original SEM	unbiased SEM	ASKKM	MKL+tag	ASKKM+tag
aeroplane	0.443	0.467	0.413	0.355	0.468	0.592	0.524
bicycle	0.182	0.233	0.146	0.148	0.331	0.324	0.361
bird	0.229	0.221	0.191	0.22	0.196	0.376	0.451
boat	0.301	0.315	0.227	0.259	0.409	0.519	0.478
bottle	0.211	0.203	0.166	0.166	0.204	0.154	0.177
bus	0.294	0.253	0.159	0.277	0.339	0.278	0.481
car	0.441	0.457	0.353	0.438	0.616	0.501	0.533
cat	0.281	0.265	0.203	0.259	0.321	0.366	0.402
chair	0.371	0.318	0.322	0.332	0.394	0.3	0.388
cow	0.162	0.175	0.135	0.159	0.199	0.117	0.159
diningtable	0.242	0.232	0.167	0.221	0.277	0.255	0.289
dog	0.308	0.279	0.213	0.313	0.316	0.331	0.401
horse	0.265	0.278	0.182	0.181	0.437	0.637	0.697
motorbike	0.257	0.228	0.148	0.251	0.361	0.383	0.428
person	0.654	0.638	0.551	0.596	0.696	0.703	0.617
pottedplant	0.174	0.175	0.162	0.183	0.191	0.212	0.186
sheep	0.208	0.212	0.145	0.163	0.169	0.218	0.297
sofa	0.218	0.251	0.171	0.181	0.266	0.191	0.201
train	0.416	0.394	0.182	0.309	0.421	0.617	0.683
TVmonitor	0.261	0.228	0.172	0.223	0.331	0.236	0.344
mAP	0.296	0.291	0.221	0.262	0.347	0.366	0.405

Table 2. Table 2: Results on MIR Flickr Dataset

MIR Flickr	without tag					with tag
	S4VM	co-training	original SEM	unbiased SEM	ASKKM	MKL+tag	ASKKM+tag
animals	0.266	0.311	0.256	0.256	0.286	0.31	0.287
baby	0.132	0.122	0.114	0.121	0.109	0.075	0.153
baby*	0.154	0.128	0.121	0.122	0.163	0.161	0.184
bird	0.131	0.136	0.125	0.131	0.126	0.124	0.139
bird*	0.127	0.131	0.121	0.119	0.143	0.163	0.217
car	0.221	0.184	0.141	0.227	0.227	0.229	0.223
car*	0.156	0.212	0.138	0.174	0.249	0.305	0.263
clouds	0.585	0.641	0.421	0.501	0.676	0.612	0.682
clouds*	0.459	0.53	0.271	0.482	0.613	0.537	0.677
dog	0.165	0.157	0.133	0.146	0.166	0.182	0.276
dog*	0.183	0.202	0.134	0.133	0.138	0.212	0.315
female	0.443	0.438	0.405	0.466	0.391	0.44	0.441
female*	0.364	0.377	0.293	0.377	0.375	0.313	0.389
flower	0.258	0.279	0.253	0.233	0.336	0.373	0.413
flower*	0.276	0.271	0.223	0.266	0.3	0.424	0.391
food	0.248	0.26	0.139	0.179	0.321	0.333	0.354
indoor	0.529	0.554	0.515	0.525	0.523	0.514	0.542
lake	0.207	0.214	0.218	0.211	0.223	0.159	0.244
male	0.401	0.417	0.315	0.423	0.346	0.366	0.385
male*	0.327	0.303	0.287	0.308	0.321	0.255	0.283
night	0.429	0.437	0.189	0.326	0.383	0.471	0.436
night*	0.304	0.301	0.135	0.211	0.321	0.368	0.426
people	0.649	0.629	0.604	0.582	0.604	0.629	0.671
people*	0.554	0.562	0.516	0.532	0.535	0.554	0.597
plant life	0.547	0.63	0.491	0.486	0.636	0.613	0.643
portrait	0.414	0.443	0.389	0.421	0.454	0.474	0.441
portrait*	0.448	0.406	0.402	0.334	0.437	0.429	0.423
river	0.194	0.205	0.184	0.195	0.218	0.234	0.295
river*	0.117	0.118	0.118	0.124	0.051	0.047	0.094
sea	0.374	0.321	0.334	0.408	0.448	0.437	0.437
sea*	0.184	0.193	0.177	0.135	0.177	0.255	0.302
sky	0.642	0.647	0.603	0.589	0.719	0.693	0.726
structures	0.658	0.652	0.602	0.651	0.659	0.655	0.693
sunset	0.374	0.368	0.226	0.342	0.416	0.543	0.487
transport	0.309	0.295	0.286	0.289	0.326	0.321	0.395
tree	0.437	0.485	0.375	0.418	0.469	0.453	0.461
tree*	0.209	0.265	0.175	0.228	0.234	0.231	0.326
water	0.428	0.437	0.396	0.451	0.495	0.452	0.513
mAP	0.339	0.348	0.285	0.319	0.359	0.367	0.401

Equations

$$\underset{\Theta }{\min }KL(P(X,Y)\Vert f(X,Y|\Theta ))$$

$$KL(P(X,Y)\Vert f(X,Y|\Theta ))=\int p(X,Y)\log \frac{p(X,Y)}{f(X,Y|\Theta )}dx$$

$$\max \{\sum _{x_i\in S_l}\log (f(x_i,y_i|\Theta ))+\sum _{x_j\in S_u}\log (f(x_j|\Theta ))\}$$

$$\underset{\Theta }{\min }\{KL(P(X,Y)\Vert f(X,Y|\Theta ))+\frac{|S_u|}{|S_l|}KL(P(X,Y)\Vert f(X|\Theta ))\}$$

$$\underset{\Theta }{\max }\{\frac{1}{1+\lambda }E[\log f(X,Y|\Theta )]+\frac{\lambda }{1+\lambda }E[\log f(X|\Theta )]\}$$

$$\underset{\Theta }{\min }KL(P(X,Y)\Vert f(X|\Theta ))$$

$$\underset{\Theta }{\min }\{KL(P(X,Y)\Vert f(X,Y|\Theta ))+KL(P(X,Y)\Vert f(X|\Theta ))\}$$

$${\mathfrak{L}}_f({\Theta }_{unsup}^{\ast })={\mathfrak{L}}_f({\Theta }_{smsup}^{\ast })\ge {\mathfrak{L}}_f({\Theta }_{usmsup}^{\ast })\ge {\mathfrak{L}}_f({\Theta }_{sup}^{\ast })$$

$$\underset{|S_u|\to \infty }{\lim }P({\mathfrak{L}}_f({\hat{\Theta }}_{smsup}^{S_l,S_u})>{\mathfrak{L}}_f({\hat{\Theta }}_{sup}^{S_l}))>0$$

$$KL(f(X,Y|{\hat{\Theta }}_{smsup})||f(X,Y|{\hat{\Theta }}_{usmsup}))$$

$$\underset{|S_u|\to \infty }{\lim }KL(f(X,Y|{\hat{\Theta }}_{smsup})||f(X,Y|{\hat{\Theta }}_{usmsup}))=0$$

$${\Theta }_{smsup}^{\ast }={\Theta }_{usmsup}^{\ast }={\Theta }_{sup}^{\ast }={\Theta }_{unsup}^{\ast }$$

$$\underset{|S_u|\to \infty }{\lim }$$

$$=$$

$$Criterion=\sum _{x_i\in S_l}I({\mathfrak{B}}_{{\hat{\Theta }}_{usmsup}}(x_i)\ne {\mathfrak{B}}_{{\hat{\Theta }}_{smsup}}(x_i))$$

$$g(c_i)=\{\begin{array}{ll}{y}_i& \text{if}{\mathfrak{B}}_{{\hat{\Theta }}_{usmsup}}(x_i)={\mathfrak{B}}_{{\hat{\Theta }}_{smsup}(x_i)}\\ {\mathfrak{B}}_{{\hat{\Theta }}_{usmsup}}(x_i)& \text{if}{\mathfrak{B}}_{{\hat{\Theta }}_{usmsup}}(x_i)\ne {\mathfrak{B}}_{{\hat{\Theta }}_{smsup}(x_i)}\end{array}$$

$$\{{\hat{\phi }}_1,{\hat{Z}}^1\}=$$

$$\sum _{x_i\in S_u}{z}_{i,k}^1\parallel {\phi }_1(x_i)-{\phi }_1({\mu }_1^k){\parallel }^2\}$$

$$\{{\hat{\phi }}_2,{\hat{Z}}^2\}=$$

$$\frac{N_l}{N_u+N_l}\sum _{x_i\in S_u}{z}_{i,k}^2\parallel {\phi }_2(x_i)-{\phi }_2({\mu }_2^k){\parallel }^2\}$$

$$\begin{array}{rl}& AP=\frac{1}{11}\sum _{r\in 0,0.1,...,1}{P}_{interp}(r)\\ & P_{interp}(r)=\underset{\tilde{r}:\tilde{r}\ge r}{\max }p\tilde{r}\end{array}$$

Full text

1 Introduction

Semi-supervised learning (SSL) plays an important role in many real world machine learning applications, such as image classification Papandreou et al. (2015), speech recognition Cui et al. (2012), and text categorization Liu et al. (2016); Sun et al. (2012), where the cost of annotating unlabeled data is generally considered too high to afford. SSL tries to make use of additional unlabeled data together with a limited amount of labeled data to enhance model learning accuracy Zhu et al. (2003), and thus has gained a lot of researchers’ attention Yang and Priebe (2011); Fox-Roberts and Rosten (2014).

However, it is generally believed that using additional unlabeled data for SSL does not always guarantee increase in model learning accuracy. Sometimes, it may end up with performance degradation. This phenomenon is known as model misspecification Yang and Priebe (2011); Fox-Roberts and Rosten (2014); Bach (2006) or safe semi-supervised learning Zhou and Li (2010). Yet in essence, model misspecification and safe semi-supervised learning are two concepts proposed independently, despite their common goal of designing SSL approach so that its performance, even in the worst case, is still better than that of the simple supervised learning approach Li et al. (2016, 2017). For safe semi-supervised learning, most of the existing approaches were proposed for non-parametric models Li et al. (2016); Li and Zhou (2011). For instance, in Li et al. (2016), they first set a baseline classifier corresponding to the supervised learners in the worst cases, and then further optimize the performance of the classifier using the proposed SSL method. The performance difference between the classifiers learned in supervised and semi-supervised manners is crucial in designing such SSL algorithms.

Approaches proposed from the perspective of model misspecification are mostly for parametric models Yang and Priebe (2011); Loog and Jensen (2015); Loog (2016) where the mismatch between the unlabeled data and the employed generative model are often used to guide the model learning. For instance, the mixture model is adopted in Yang and Priebe (2011) to represent both labeled and unlabeled data, which then formulates the corresponding Bayes plug-in classifier based on the mixture density functions. The performance of this Bayes plug-in classifier seriously relies on how far the learnt parameters are from the true ones. The performance degradation mainly comes from the model bias and estimation error. While how the unlabeled data could affect the SSL model performance is theoretically analysed in Yang and Priebe (2011), how to detect the onset of the model misspecification and how to solve the challenge have not been addressed yet. Along this line, the lower bound and upper bound of the performance of semi-supervised generative models are analyzed in Fox-Roberts and Rosten (2014), and then the authors propose to use the ratio of unlabeled data to labeled data to control the model performance. Inspired by these two works, we propose a generative model based approach for addressing the model misspecification issue. Different from the two aforementioned methods, we not only explore how to detect whether the model misspecification occurs, but also propose a model modification method instead of controlling the unlabeled data to be utilized.

The remaining of this paper is organized as follows. We first present related work in Section 2 and formulate the model misspecification problem using generative models in Section 3. The proposed SSL learning approach is described in Section 4. Experimental results based on two image classification datasets are reported in Section 5. We conclude the paper in Section 6.

2 Related Work

The difficulty to achieve a reliable semi-supervised learning method has been reported in some earlier works (e.g., Cozman et al. (2002)). The general believe is that a better SSL model is not always guaranteed even though additional unlabeled data are used for the model learning. There exist a number of factors that may affect the SSL performance such as the quality of the training data as well as the classifier itself Yang and Priebe (2011). Some researchers considered that to be the consequence of a wrong model assumption Wang et al. (2012). Practically, it is hard to assume a perfect generative model without any prior knowledge about the unknown data set. In Zhou and Li (2010); Li et al. (2017), the underlying challenge is viewed as having some unlabeled data assigned with incorrect labels which are then used to augment the labeled training data set.

Some recent research works proposed safety-aware mechanisms to restrict the SSL from using risky unlabeled data Wang and Chen (2013); Li et al. (2017, 2016). Other than the earlier disagreement based SSL Bennett et al. (1998), several safe SSL approaches have been proposed, such as S3VMs Bennett et al. (1998), S4VM Li and Zhou (2011) and UMVP Li et al. (2016) with promising experimental results. However, most of them fall into the non-parametric category. To leverage on their good generalization capabilities, generative models based on SSL are common for many applications such as image classification Yang et al. (2017). The two most representative works addressing the model misspecification issue include Yang and Priebe (2011) and Fox-Roberts and Rosten (2014). In Yang and Priebe (2011), they started with the asymptotic optimal parameters of two generative models obtained using fully supervised learning and fully unsupervised learning, respectively. Then, they proved that if the KL divergence between the distributions generated by the two generative models is small, the SSL performance is less likely to be affected by the addition of unlabeled data. More theoretical analysis was provided in Fox-Roberts and Rosten (2014) where the local and global bounds on the divergence of SSL are formulated. Then, they proposed an unbiased generative SSL which is defined based on an unbiased likelihood estimator exponentially controlled by the ratio of the number of labeled data over the total number of data.

Our work is different from Yang and Priebe (2011) and Fox-Roberts and Rosten (2014) as follows. We focus more on adaptively modifying the structure of the generative models instead of controlling risky data to be used. Furthermore, we propose a criterion to determine whether a model misspecification occurs or not. To the best of our knowledge, neither approaches have been considered in the literature.

3 Model Misspecification Problem

Given a finite number of labeled data and an infinite number of unlabeled data, it is impossible to directly detect whether the model misspecification occurs or not as the true data distribution is generally unknown. Figure 1 gives an illustrating example of the proposed approach. Assume that the model estimation error is ignored. In Figure 1 (a), the assumed semi-supervised generative model (SEM) is misspecified, and its performance, in the worst case, would converge to the classification loss bound $L_{unsup}^{*}$ of SEM learnt in an unsupervised manner if infinite unlabeled data were used. And the unbiased SEM Fox-Roberts and Rosten (2014) would converge to the classification loss bound $L_{sup}^{*}$ of SEM learnt in the supervised manner. However, these classification loss bounds are higher than the best classification loss $L_{opt}^{*}$ obtained by a well-specified model. Therefore, the minimum model bias from the learnt SEMs to the true data model can be approximated by $L_{opt}^{*}-L_{sup}^{*}$ which is indicated by the loss difference, i.e., $L_{sup}^{*}-L_{unsup}^{*}$. Such loss difference can be approximated by the KL distance between two SEMs. If the assumed SEM is well-specified as plotted in Figure 1 (b), the classification loss bound of the SEM, unbiased SEM and the well-specified SEM will be the same, i.e., $L_{opt}^{*}=L_{sup}^{*}=L_{unsup}^{*}$, in the ideal case. Consequently, the model difference (KL) of two SEMs should be also small. Inspired by this observation, we conjecture that there must exist the model misspecification if the KL between two SEMs is getting large and therefore we can use this value to determine whether a model misspecification occurs or not. The problem is formulated in the following paragraphs.

Let $X:=(x_{1},x_{2},\cdots,x_{N})^{T}$ and $Y:=(y_{1},y_{2},\cdots,y_{N})^{T}$ denote the training data and the corresponding label, respectively. The set of labeled data and unlabeled data are denoted as $S_{l}:=(X,Y)$ and $S_{u}:=(x^{U}_{1},x^{U}_{2},\cdots)^{T}$, respectively. The mixed set of labeled and unlabeled data is denoted as $D=S_{l}\bigcup S_{u}$. Suppose $P(X,Y)$ is the true data distribution for $D$ and $f(X,Y|\Theta)$ denote the assumed generative model with its parameter set as $\Theta$. A supervised generative model is obtained by learning the model parameters that best fit $P(X,Y)$, written as

[TABLE]

where $KL(\cdot)$ is the Kullback-Leibler divergence, defined as

[TABLE]

Given a generative model $f$ (e.g. GMM) with its parameter set $\Theta$, a popular semi-supervised learning objective function is given as,

[TABLE]

Theorem 1**.**

If $|S_{l}|\rightarrow\infty$, $|S_{u}|\rightarrow\infty$, SSL problem in Eq.3 is equivalent to

[TABLE]

Proof. For brevity, let $N_{l}=|S_{l}|$,$N_{u}=|S_{u}|$, $\lambda=N_{u}/N_{l}$, following the Theorem proposed in Cozman et al. (2003), the MLE problem in Eq.3 is stated as:

[TABLE]

Using Eq.2, Eq.5 is equivalent to Eq.4.

Corollary 1**.**

If $\lambda\rightarrow\infty$, SSL problem as Eq. 4 degenerates back to an unsupervised learning problem.

[TABLE]

For this reason, Fox-Roberts and Rosten (2014) proposed a weighting strategy to avoid the model misspecification issue, given as

[TABLE]

The effectiveness of this strategy is quite tricky as it seriously relies on whether there are enough labeled data or not. It becomes risky when the labeled data are few. Therefore, we propose a safer strategy as follows.

For a generative model $f$, let $\Theta_{sup}$ denote the parameter set learnt in a supervised manner using Eq. 1. Similarly, $\Theta_{smsup}$, $\Theta_{usmsup}$, and $\Theta_{unsup}$ are the solutions to Eqs. 4, 7, and 6, respectively, denoting original SSL models, unbiased SSL models, and unsupervised models. As the data set only contains a finite number of data points, the best estimations ${\Theta}^{*}_{sup}$,${\Theta}^{*}_{smsup}$,${\Theta}^{*}_{usmsup}$, and ${\Theta}^{*}_{unsup}$ are only theoretical values. On the finite data set $D$, $\hat{\Theta}_{smsup}^{S_{l},S_{u}}$, $\hat{\Theta}_{usmsup}^{S_{l},S_{u}}$ are the solutions to Eqs. 4 and 7 respectively. Let $\mathfrak{L}_{f}(\Theta)$ denote the loss of the Bayes plug-in classifier.

Corollary 2**.**

When $N_{l}\rightarrow\infty$, $N_{u}\rightarrow\infty$ and $N_{l}/N_{u}\rightarrow 0$, for the ideal solutions ${\Theta}^{*}_{sup}$,${\Theta}^{*}_{smsup}$,${\Theta}^{*}_{usmsup}$, and ${\Theta}^{*}_{unsup}$, there is,

[TABLE]

Thus, if $f$ is incorrect, the difference between $\hat{\Theta}_{usmsup}$ and $\hat{\Theta}_{smsup}$ will become larger and larger when more unlabeled data are used for model learning.

Definition 1**.**

With Corollay 2, if $\mathfrak{L}_{f}({\Theta}_{unsup}^{*})>\mathfrak{L}_{f}({\Theta}_{sup}^{*})$, then $f$ is misspecified.

Theorem 2**.**

if $\mathfrak{L}_{f}(\Theta^{*}_{unsup})>\mathfrak{L}_{f}(\Theta^{*}_{sup})$, then $\exists$ $S_{l}$, s.t.

[TABLE]

This theorem shows that the semi-supervised learning yields degradation with a positive probability when more unlabeled data are introduced. The proof is straightforward and is not given due to the page limitation. By optimizing Eqs 3 and 7, two distributions $f(X,Y|{\hat{\Theta}}_{smsup})$ and $f(X,Y|{\hat{\Theta}}_{usmsup})$ can be acquired. Then, the difference between the original and the unbiased semi-supervised learning can be defined as

[TABLE]

Theorem 3**.**

If the model $f$ with $\Theta$ is not misspecified,

[TABLE]

Proof. By Definition 1, if $f$ is not misspecified, there is $\mathfrak{L}_{f}({\Theta}_{unsup}^{*})=\mathfrak{L}_{f}({\Theta}_{sup}^{*})$. Considering $\mathfrak{L}_{f}({\Theta}_{unsup}^{*})\geq\mathfrak{L}_{f}({\Theta}_{smsup}^{*})\geq\mathfrak{L}_{f}({\Theta}_{sup}^{*})$ and $\mathfrak{L}_{f}({\Theta}_{unsup}^{*})\geq\mathfrak{L}_{f}({\Theta}_{usmsup}^{*})\geq\mathfrak{L}_{f}({\Theta}_{sup}^{*})$, there is,

[TABLE]

So,

[TABLE]

With the proposed theorems and corollaries, we have theoretically proved that the correctness of the previous conjectures illustrated in Figure 1 and a well-specified semi-supervised model-based classifier is proposed in the next Section.

4 The Proposed ASKKM Approach

To alleviate the model misspecification problem, we propose to adapt the model structure. In particular, we focus on kernel $k$-means model which can be considered as a special case of Gaussian mixture models (GMM), and explore mechanisms to learn the model structure and the model parameters to ensure the model to be well-specified. Although different model complexity measures (such as BIC Watanabe (2013)) have been proposed to determine the optimal generative models like GMM, these measures were designed primarily for density estimation, and thus cannot be directly applied in the semi-supervised setting to address the misspecification problem. As discussed in Section 3, we adopt the KL divergence between the original and unbiased semi-supervised learning to guide the adaptive model structure learning of a kernel $k$-means model. This section presents the proposed adaptive model modification based semi-supervised kernel $k$-means model (ASKKM for short).

4.1 Model Misspecification Criterion

As illustrated in Figure 1, if the assumed model is misspecified, the original SEM and unbiased SEM converge to different classification loss bound. The difference between classification loss bounds is approximated by the KL divergence between two SEMs. Practically, the discrete KL divergence might be problematic when calculated on the limited number od data points. Therefore, the aggregated classification disagreement is adopted to approximate the bound difference. If the aggregated classification disagreement is large enough, then their KL divergence must be greater than 0 and thus exists model misspecification according to Theorem 3. Denote $\mathfrak{B}_{\hat{\Theta}_{usmsup}}(x)$ and $\mathfrak{B}_{\hat{\Theta}_{smsup}}(x)$ as the Bayes plug-in classifiers for original SEM and unbiased SEM, respectively. The criterion for model misspecification is defined as

[TABLE]

where $I(\cdot)$ is the indicator function. If $Criterion$ is greater than a predefined threshold $\epsilon$, the corresponding assumed model is determined as misspecified.

If model misspecification occurs, we gradually increase the model complexity of the employed semi-supervised generative model by modifying K, i.e., the number of components. Specially, for each labeled training data $x_{i}\in S_{l}$, a new label $c_{i}$ is assigned to it if $Criterion>\epsilon$. The size of new label set $C$ is then larger than that of the given class label set $Y$, i.e., $|C|>|Y|$. For the classification task, a mapping function from new label set to the given label set is defined as,

[TABLE]

With this function, a new cluster is introduced for the new class label and thus the model structure is adaptively modified.

4.2 Adaptive Semi-supervised Kernel K-means Model

For the classification task, a kernel $k$-means is adopted in the paper, and accordingly Eqs. 4 (original SEM) and Eq.7 (unbiased SEM) can be rewritten respectively as

[TABLE]

[TABLE]

where $\varphi_{1},\mu_{1}^{k}$ and $\varphi_{2},\mu_{2}^{k}$ are respectively the kernel maps and the centroids for the original and weighted semi-supervised kernel $k$-means, and $Z^{1}$ and $Z^{2}$ are the cluster assignments with $Z^{1}=Z^{1}_{l}\cup Z^{1}_{u}$ and $Z^{2}=Z^{2}_{l}\cup Z^{2}_{u}$. That is, if $x_{i}$ is assigned to the $k$-th cluster according to Eq. 16 or 17, $z^{1}_{i,k}=1$ or $z^{2}_{i,k}=1$. Intuitively speaking, the proposed approach tracks the difference between $Z_{l}^{1}$ and $Z_{u}^{2}$. If $Z_{l}^{1}\neq Z_{u}^{2}$, Eq. 14 is used to check whether model misspecification occurs or not. Details of the proposed ASKKM is illustrated in Algorithm 1.

5 Experimental Results

For experimental evaluation, we evaluate the proposed ASKKM using two image classification data sets, i.e., PASCAL VOC’07 Everingham (2010) and MIR Flickr Huiskes and Lew (2008). PASCAL VOC’07 consists of 9,963 images from 20 classes and 804 annotated tags. Among them, 5,011 images are selected as the training set and the rest form the test set. MIR Flikr contains 25,000 images and 457 tags from 38 classes collected from the Flickr website. Among them, 12,500 images are randomly selected for training and the rest form the test set. For image feature representations, two local features (SIFT, Hue), three global histogram features (RGB, Hsv and Lab) as well as GIST are used to represent each image. We adopted different distance metrics for features of different types. In particular, the Manhattan distance, Euclidean distance and Chi-square distance are used respectively for the histogram features, the GIST features and the local features. The state-of-the-art SSL algorithms as well as semi-supervised generative models are chosen for model comparison, including S4VM Li and Zhou (2011), co-training Zhou and Li (2005), semi-supervised EM (SEM) Fox-Roberts and Rosten (2014) and MKL Guillaumin et al. (2010). In addition to utilizing the labels of images, the original MKL also utilizes the tags of images. Therefore, we extend the proposed ASKKM in the same way as what MKL does to utilize the tag information. For performance evaluation metric, we adopt average precision (AP) which is the evaluation criterion used in PASCAL VOC competition, written as

[TABLE]

where this criterion requires the recall $r$ to take value from 0 to 1 with the step as 0.1, and then sums up the precision over all $r$ and takes the average value. The performance comparison results are reported in the following sections.

5.1 Verification of Model Structure Modification

To evaluate how the adaptive modification on model structure could affect the model performance, experiments are performed on “car” data set of PASCAL VOC’07. For this binary classification task, $K$ is fixed to 2 for the original SEM and ubiased SEM model Fox-Roberts and Rosten (2014). However, the true data distributions for image data may contain more than two clusters (components). Therefore, the proposed ASKKM adaptively modifies $K$ once the model misspecification is detected during the experiments. The comparison results are then plotted in Figure 2.

From this figure, it is noticed that the model performance of the original SEM gradually degrades after $N_{u}>100$. The unbiased SEM is better than the original SEM as its AP value, although slightly fluctuates, almost keeps around 0.4 which does not degrade. For the performance of the ASKKM, it slowly increases at the beginning part of the curve. After $N_{u}>100$, its AP value dramatically increases where model misspecification is detected and the model structure is accordingly modified. Then, the ASKKM gradually converge with the addition of unlabeled data. The converged model performance of ASKKM is much better than that of the compared semi-supervised generative models. This verifies that a well-specified semi-supervised models could acquire a superior model performance.

5.2 Model Performance Comparison

The model performance evaluation is performed on aforementioned two image data sets. For each class in the data set, we only choose few labeled data, i.e., $N_{l}=20$ for PASCAL VOC’07 data set and $N_{l}=50$ for MIR Flikr data set. We not only compare our approach with two semi-supervised generative models but also with the most representative semi-supervised learning algorithms such as S4VM and co-training. Experimental results are reported in Table 1 and Table 2.

From Table 1, the model performance of the ASKKM is the best on 17 classes out of 20 classes when tag information is not considered. The S4VM achieves the best AP value in class “aeroplane” and “bottle”. The MAP value of the ASKKM is $17.2\%$ higher than the second best model S4VM. It is also noticed that semi-supervised SVM based algorithms performs better than the generative model based ones. The superior performance of the proposed approach indicates the superiority of the ASKKM over the rest approaches. If tag information is considered, it is observed that MKL+tag is better than these approaches without the integration of tag information, and this is consistent with our intuition. However, the ASKKM+tag is better than MKL+tag in 16 classes and the overall MAP of the ASKKM is $10.6\%$ higher than that of MKL+tag. This further verify the effectiveness of the proposed approach. Similar observations could be found in the evaluation results on MIR Flickr data set reported in Table 2. From these rigorous experimental results, we can conclude the proposed ASKKM is superior to the state-of-the-art semi-supervised learning approaches in terms of average precision and mean average precision.

6 Conclusion

To learn a reliable semi-supervised models is of utmost importance. Most of existing works are non-parametric based ones and the generative model based approach is seldom studied. This paper first proposes a criterion to judge whether a model misspecificatoin occurs or not. Then an adaptive semi-supervised kernel K-means model (ASKKM) is proposed for the model misspecified problem. At last, we rigorously evaluate the proposed ASKKM on two image classification data sets, i.e., PASCAL VOC’07 and MIR Flickr. Promising results demonstrate the efficacy of the proposed approach.