Transfer feature generating networks with semantic classes structure for zero-shot learning

TL;DR

This paper introduces TFGNSCS, a transfer feature generating network that leverages semantic class structures and transfer learning to improve zero-shot and generalized zero-shot learning performance.

Contribution

The paper proposes a novel transfer feature generating network that incorporates semantic class structures and transfer learning to enhance ZSL and GZSL accuracy.

Findings

Outperforms state-of-the-art methods on four datasets

Effectively models semantic class relationships

Improves feature generation for unseen classes

Abstract

Feature generating networks face to the most important question, which is the fitting difference (inconsistence) of the distribution between the generated feature and the real data. This inconsistence further influence the performance of the networks model, because training samples from seen classes is disjointed with testing samples from unseen classes in zero-shot learning (ZSL). In generalization zero-shot learning (GZSL), testing samples come from not only seen classes but also unseen classes for closer to the practical situation. Therefore, most of feature generating networks difficultly obtain satisfactory performance for the challenging GZSL by adversarial learning the distribution of semantic classes. To alleviate the negative influence of this inconsistence for ZSL and GZSL, transfer feature generating networks with semantic classes structure (TFGNSCS) is proposed to construct networks model for improving the performance of ZSL and GZSL. TFGNSCS can not only consider the semantic structure relationship between seen and unseen classes, but also learn the difference of generating features by transferring classification model information from seen to unseen classes in networks. The proposed method can integrate the transfer loss, the classification loss and the Wasserstein distance loss to generate enough CNN features, on which softmax classifiers are trained for ZSL and GZSL. Experiments demonstrate that the performance of TFGNSCS outperforms that of the state of the arts on four challenging datasets, which are CUB,FLO,SUN, AWA in GZSL.

Tables

Table 1. Table 1: Datasets statistics involve semantic embedding ℂ ℂ \mathbb{C} (att/dimension for attribute per class or stc/dimension for sentences), number of classes in training classes ( 𝕐 s subscript 𝕐 𝑠 \mathbb{Y}_{s} includes training + validation) and test classes ( 𝕐 u ) \mathbb{Y}_{u}) , visual feature 𝕏 𝕏 \mathbb{X} in experiments.

Datasets	$ℂ$	$\left|{𝕐}_s\right|+\left|{𝕐}_u\right|$	$\left|{𝕐}_s\right|$	$\left|{𝕐}_u\right|$	$𝕏$
AwA	att/85	$50$	$27+13$	$10$	Deep feature/2048 by ResNet[45]
CUB	att/312	$200$	$100+50$	$50$	Deep feature/2048 by ResNet[45]
SUN	att/102	$717$	$580+65$	$72$	Deep feature/2048 by ResNet[45]
FLO	stc/1024[44]	$102$	$62+20$	$20$	Deep feature/2048 by ResNet[45]

Table 2. Table 2: Comparison of TFGNSCS method with state of the art methods for ZSL with semantic feature and ResNet visual feature.

	ZSL
Method	AwA	CUB	SUN	FLO
DVN[46]	$67.7$	$57.8$	$62.4$	NA
HM[47]	$67.4$	$52.6$	$61.7$	NA
VCL[48]	$61.5$	$59.6$	$59.4$	NA
TVN[49]	$68.8$	$58.1$	$60.7$	NA
FGN[1]	$68.2$	$57.3$	$60.8$	$67.2$
TFGNSCS	70.5	60.2	64.8	68.1

Table 3. Table 3: Comparison of TFGNSCS method with state of the art methods for GZSL with semantic feature and ResNet visual feature. tr=average per-class Top-1 accuracy (%) on seen classes, ts=average per-class Top-1 accuracy (%) on unseen classes, H=harmonic mean for GZSL are reported based on the same data configurations in the different datasets splits.

	GZSL
	AwA			CUB			SUN			FLO
Method	ts	tr	H	ts	tr	H	ts	tr	H	ts	tr	H
DVN[46]	$34.9$	$73.4$	$48.5$	$26.2$	$55.1$	$35.5$	$25.3$	$34.6$	$29.2$	NA	NA	NA
HM[47]	$38.7$	$74.6$	$51.0$	$31.5$	$40.2$	$35.3$	$41.2$	$26.7$	$32.4$	NA	NA	NA
VCL[48]	$21.4$	89.6	$34.6$	$15.6$	86.3	$26.5$	$10.4$	63.4	$17.9$	NA	NA	NA
TVN[49]	$27.0$	$67.9$	$38.6$	$26.5$	$62.3$	$27.2$	$22.2$	$38.3$	$28.1$	NA	NA	NA
FGN[1]	$57.9$	$61.4$	$59.6$	$43.7$	$57.7$	$49.7$	$42.6$	$36.6$	$39.4$	$59.0$	$73.8$	$65.6$
TFGNSCS	60.3	$69.5$	64.5	47.6	$59.7$	53.0	44.3	$37.5$	40.6	61.1	78.6	68.7

Table 4. Table 4: Comparison of TFGNSCS method with the base-line methods for ZSL with semantic feature and ResNet visual feature.

	ZSL
Method	AwA	CUB	SUN	FLO
FGN[1]	$68.2$	$57.3$	$60.8$	$67.2$
TFGNSCS-1	$69.4$	$59.5$	$63.7$	$67.8$
TFGNSCS-2	$68.9$	$58.6$	$62.9$	$67.3$
TFGNSCS	70.5	60.2	64.8	68.1

Table 5. Table 5: Comparison of TFGNSCS method with the base-line methods for GZSL with semantic feature and ResNet visual feature. tr=average per-class Top-1 accuracy (%) on seen classes, ts=average per-class Top-1 accuracy (%) on unseen classes, H=harmonic mean for GZSL are reported based on the same data configurations in the different datasets splits.

	GZSL
	AwA			CUB			SUN			FLO
Method	ts	tr	H	ts	tr	H	ts	tr	H	ts	tr	H
FGN[1]	$57.9$	$61.4$	$59.6$	$43.7$	$57.7$	$49.7$	$42.6$	$36.6$	$39.4$	$59.0$	$73.8$	$65.6$
TFGNSCS-1	$58.4$	69.8	$63.5$	$46.7$	$58.4$	$51.8$	$43.8$	37.7	$40.5$	$60.1$	$77.6$	$67.7$
TFGNSCS-2	$58.2$	$67.9$	$62.6$	$45.4$	$59.6$	$51.5$	$43.0$	$37.1$	$39.8$	$59.8$	$77.2$	$67.3$
TFGNSCS	60.3	$69.5$	64.5	47.6	59.7	53.0	44.3	$37.5$	40.6	61.1	78.6	68.7

Table 6. Table 6: Comparison of TFGNSCS method with the alternative transfer method for ZSL with semantic feature and ResNet visual feature.

	ZSL
Method	AwA	CUB	SUN	FLO
TFGNSCS-alt	$69.7$	60.8	$63.4$	68.3
TFGNSCS	70.5	$60.2$	64.8	$68.1$

Table 7. Table 7: Comparison of TFGNSCS method with the alternative transfer method for GZSL with semantic feature and ResNet visual feature. tr=average per-class Top-1 accuracy (%) on seen classes, ts=average per-class Top-1 accuracy (%) on unseen classes, H=harmonic mean for GZSL are reported based on the same data configurations in the different datasets splits.

	GZSL
	AwA			CUB			SUN			FLO
Method	ts	tr	H	ts	tr	H	ts	tr	H	ts	tr	H
TFGNSCS-alter	60.5	$68.9$	$64.4$	$46.8$	59.9	$52.5$	$43.9$	37.8	40.6	$60.7$	78.8	$68.5$
TFGNSCS	$60.3$	69.5	64.5	47.6	$59.7$	53.0	44.3	$37.5$	40.6	61.1	$78.6$	68.7

Equations

$$\begin{array}{rl}& L_{WGAN}=E[D(x_s,c(y_s))]-E[D({\tilde{x}}_s,c(y_s))]-\lambda E[(\Vert {▽}_{\hat{x}}D(\hat{x},c(y_s)){\Vert }_2-1{)}^2],\end{array}$$

$$\begin{array}{rl}& L_{CLS}=-E_{{\tilde{x}}_s\sim p_{{\tilde{x}}_s}}[\log P(y_s|{\tilde{x}}_s;\theta )],\end{array}$$

$$\begin{array}{rl}& L_{TRA1}=-E_{{\tilde{x}}_u\sim p_{{\tilde{x}}_u}}[\log Q(y_u|{\tilde{x}}_u;\theta )],\end{array}$$

$$\begin{array}{rl}& L_{TRA2}=-E[D({\tilde{x}}_u,c(y_u))],\end{array}$$

$$\begin{array}{rlr}& \underset{G}{\min }\underset{D}{\max }L=& \underset{G}{\min }\underset{D}{\max }{L}_{WGAN}+\beta L_{CLS}+\gamma L_{TRA1}+\eta L_{TRA2},\end{array}$$

$$\begin{array}{rl}& \underset{\theta }{\min }-\frac{1}{N_{\mathbb{T}}}\sum _{(x_s,y_s)\in \mathbb{T}}\log P(y_s|x_s;\theta ),\end{array}$$

$$\begin{array}{rl}& Q(\theta )=P(\theta )W_{ss}^{-1}{W}_{su}{W}_{uu},\end{array}$$

$$\begin{array}{rl}& d(a,b)=\frac{a\cdot b}{\Vert a\Vert \cdot \Vert b\Vert },\end{array}$$

$$W_{ss}(i,j)=\{\begin{array}{rl}& d(c(y_s{)}_i,c(y_s{)}_j),c(y_s{)}_i\in N_{c(y_s{)}_j}\\ & 0\text{mbox}else,\end{array}$$

$$W_{uu}(i,j)=\{\begin{array}{rl}& d(c(y_u{)}_i,c(y_u{)}_j),c(y_u{)}_i\in N_{c(y_u{)}_j}\\ & 0\text{mbox}else,\end{array}$$

$$W_{su}(i,j)=\{\begin{array}{rl}& d(c(y_s{)}_i,c(y_u{)}_j),c(y_s{)}_i\in N_{c(y_u{)}_j}\\ & 0\text{mbox}else,\end{array}$$

$$\begin{array}{rl}& \underset{\varphi }{\min }-\frac{1}{N_{\mathbb{T}}}\sum _{(x,y)\in \mathbb{T}}\log P(y|x;\varphi ),\end{array}$$

$$\begin{array}{rl}& f(x)=\arg \max P(y|x;\varphi ),\end{array}$$

$$\begin{array}{rl}& Q(\theta )=T(P(\theta ))=P(\theta )(I-W_{ss}{)}^{-1}{W}_{su},\end{array}$$

Full text

††journal: Journal of LaTeX Templates

1 Introduction

Based on large amounts of labeled data training, deep learning can capture the various patterns of data for large-scale recognition problem. However, in many practical application, we usually lack annotated data, which needs lots of time-consume to manually annotate. Therefore, data generation [2] [3] [4] [5] [1] with labels has become an important method for obtaining enough annotated data. Generative adversarial networks (GAN) [2] can synthesize the approximate images on object classes[3][5], but can not generate sufficiently discriminative images or features without classification information. Especially, because training samples from seen classes is disjointed with testing samples from unseen classes in ZSL or GZSL, generative features for different classes don’t accurately match with the respective distribution in GAN. In other words, there is some data shift between generative features for unseen classes and their real distribution, since generate networks model is often trained by the samples of seen classes. Existing CLSWGAN[1] considers the classification loss of seen classes for improving the performance of ZSL or GZSL. However, the classification loss of unseen classes is also important for ZSL or GZSL. Therefore, our motivation is how to transfer classification information from seen to unseen classes to construct the classification loss of unseen classes (this loss is called transfer loss) for learning generate networks model.

ZSL[6][7][8][9][10] is an arguable problem about the extreme condition of few samples. Some classes (seen classes) have visual samples , while others (unseen classes) have no visual samples during training in ZSL. In this work, we focus on the transferability of generative adversarial model, and expect to use transfer information to process the generating feature inconsistence of the unbalance learning (In Fig 1, we explain this point, which means learning model from seen classes to generate features for unseen classes)between seen and unseen classes for constructing learning model in ZSL and GZSL. Generative features for unseen classes by transfer generative adversarial model is used for the traditional supervised learning to solve ZSL and GZSL. Therefore, the main contribution in our paper is the proposed TFGNSCS based on existing CLSWGAN[1] to find the importance of transfer information for processing the unbalanced learning between seen and unseen classes. Especially, we look into the influence of the different transfer loss for generating features in ZSL or GZSL. We mainly discuss two transfer losses. One (the detail is defined by equation (3) in section 3.1) is the consideration of the structure relationship between seen and unseen classes by structure propagation (the details in section 2.2), and the other (the detail is defined by equation (4) in section 3.1)is balancing the difference of generating features between seen and unseen classes by discriminator information. In this motivation, we proposed a novel generative feature GAN method-namely TFGNSCS that is learned with a novel transfer loss improving over existing GAN-models for generating features.

Our contributions have three point as follows.(a) We present a novel adversarial generative model TFGNSCS that synthesizes CNN features of classes by optimizing and balancing the related losses, which are the transfer loss,the classification loss and the Wasserstein distance loss. (b) In four challenging datasets with the different size or granularity, the proposed TFGNSCS outperforms the state of the arts in GZSL setting. (c) Our model is generalized to different transfer information ways for evaluating the performance of generative features model.

2 Related Works

The related works of the proposed method involve generative adversarial networks(GAN), structure propagation, zero-shot learning (ZSL) and generalization gero-shot lLearning (GZSL).

2.1 Generative Adversarial Networks

GAN[2] can initially learn a generative model to follow an arbitrary distribution by a discriminative model adjustment, for example images distribution fitting. In terms of GAN theory development, this process involves three aspects. The first aspect is GAN training improvement by additional information, such as deep convolution neural network in DCGAN [11], the style and structure networks in improved DCGAN [12], and the mutual information between the latent variables and the generator distribution in InfoGAN [13]. The second aspect is conditional GAN by feeding the related information into networks, for instance class label [14] and sentence descriptions [3]. The third aspect is about stability GAN training by the relevance constraint [15], which can be Wasserstein distance [16] or Lipschitz constraint [4]. Recently, CLSWGAN [1] can utilize WGAN idea with classification loss for generating image feature, and demonstrate the promising results in ZSL and GZSL.

In this paper, we intuitively find that feature generated by the state of art CLSWGAN [1](this model only is trained by the samples of seen classes) are not enough fitting the distribution of unseen classes for learning a classifier. Hence, we present a novel GAN framework to synthesize CNN features to learn a discriminative classifier for ZSL and GZSL. Integrating the promising CLSWGAN [1] loss and transfer loss which transmits the information of the different classes to generate the discriminative feature, our proposed GAN framework outperforms CLSWGAN [1] owing to the regularizer of transfer loss.

2.2 Structure Propagation

To best of our knowledge, structure propagation is firstly proposed for image completion as a global optimization problem by enforcing structure and consistency constraints [17]. Structure can be defined as the graph structure among data samples and plays a very important role for visual information discrimination. In recent works, there are two kinds of the impressive methods. One is structure information propagation in label space, such as dynamic structure fusion and label propagation to refining the relation of objects for semi-supervised multi-modality classification [18] and information propagation mechanism from the semantic label space, which can be applied to model the interdependencies between seen and unseen class labels [19]. The other is structure information propagation between seen and unseen classes, for instant structure fusion and propagation to update the relevance of multi-semantic classes by the iteration computation for ZSL [20] [10], structure propagation constraining the encoder-decoder mechanism of the bidirectional projection for ZSL [21], and absorbing Markov chain process propagation constructing semantic class prototype graph for ZSL [22]. Although those papers have shown the information transferability of structure propagation, structure propagation is not used for generating feature in adversarial mechanism to balance the difference between seen and unseen classes.

In this paper, we expect to construct a novel GAN framework by transfer loss adding into CLSWGAN [1]. Transfer loss includes two parts. One is balancing transfer information by generative model iteration, and the other is structure propagation for accurately computing classification loss of all classes. In contrast, CLSWGAN [1] only calculates classification loss of seen classes. Therefore, structure propagation can further extend CLSWGAN [1] model to relieve the generating feature inconsistence of seen classes training model for following the distribution of unseen classes.

2.3 Zero-shot Learning

In ZSL, seen classes of model learning in training and unseen classes of model evaluation in testing are disjoint [23]. According to the utilization of deep network framework in ZSL, ZSL methods can be divided into two categories to bridge the gap between seen and unseenclasses. One involves non-deep network framework for ZSL, such as semantic attribute classifiers learning[24] [25] [6], seen class proportions combining unseen class [26] [27] [28] [8], and the learning compatibility between images and classes [29] [30] [31] [32] [33] [9]. The other category utilizes deep network framework for ZSL, for instance DeViSE model[34],latent discriminative feature learning (LDF) model [35] and quasi-fully supervised learning (QFSL) model by deep network optimizing the visual or semantic model, synthesizing example [36] or preserving semantic relation [37] by autoencoder architecture, multi-label zero-shot learning (ML-ZSL) [19] or graph convolutional network for zero-shot learning [38] with the benefit of knowledge graph,and semantics-preserving adversarial embedding network (SP-AEN) [39] or feature generating network [40] [1] based on generation adversarial mechanism.

In summary, generation adversarial frameworks demonstrate the promising results, and especially visual feature generation outperforms image generation based on the same adversarial frameworks for ZSL. However, these frameworks rarely consider transfer information based on structure propagation for finding the more discriminative feature in ZSL. Therefore, for considering the transfer loss constrains,we expect to construct a novel generation adversarial frameworks to capture discriminative information for tackling ZSL or GZSL.

3 Transfer Feature Generating Networks

Existing ZSL methods only use the information of seen classes (label data, image feature data and semantic data) during training, and predict the label of unseen classes by the potential relation of semantic space (In a complete semantic space, semantic concepts have an uniform description, on which their distribution relation can be captured.). The main idea of the proposed model transfers the information of seen classes into synthesized feature of unseen classes by structure propagation, and iteratively constructs the learned model based on the real information of seen classes and the generative information of unseen classes. Therefore, the key point of the proposed method not only draws support from semantic embedding vector but also models transfer relation (structure propagation) to generate CNN features without any images of the class. The transfer relation can alleviate the inconsistence of the generation distribution on all categories. Because we can use synthesized CNN features as samples of unseen classes, ZSL can be converted into supervised learning. We can train Softmax classifier for recognizing unseen classes.

We define the following notation for describing ZSL or GZSL. $S=\{(x_{s},y_{s},c(y_{s}))|x_{s}\in\mathbb{X},y_{s}\in\mathbb{Y}_{s},c(y_{s})\in\mathbb{C}\}$ is training set, which includes seen classes. $x_{s}\in\mathbb{R}^{d_{x}}$ is the CNN feature with $d_{x}$ dimension in $\mathbb{X}$ feature sets, $y_{s}$ stands for the class label of $x_{s}$ in $\mathbb{Y}_{s}=\{y_{s}|s=1,2,...,K\}$, and $c(y_{s})\in\mathbb{R}^{d_{c}}$ denotes the class $y_{s}$ of the semantic embedding, which represents the class vector of semantic description(such as attributes), in $\mathbb{C}$ semantic embedding sets.In addition, the available information of unseen classes in training is $U=\{(y_{u},c(y_{u}))|y_{u}\in\mathbb{Y}_{u},c(y_{u})\in\mathbb{C}\}$ where $y_{u}$,$c(y_{u})$ and $\mathbb{Y}_{u}=\{y_{u}|u=1,2,...,M\}$ is respectively the class label, the class embedding and the class label set in unseen classes without image and feature. Therefore, the purpose of ZSL is to learn a projection $f:\mathbb{X}\times\mathbb{C}\rightarrow\mathbb{Y}_{s}$ for discriminating images of unseen classes belonging to which one in $\mathbb{Y}_{u}$, while the task of GZSL learns the same projection for recognizing seen and unseen classes of images being which one of $\mathbb{Y}_{s}\bigcup\mathbb{Y}_{u}$, where $\mathbb{Y}_{s}\bigcap\mathbb{Y}_{u}=\varnothing$.

3.1 Feature generation

In this section, we discuss the feature generation model CLSWGAN [1] based on GAN framework as the proposed model basis. The main idea of GAN is playing a game between a generative network $G$ and a discriminative $D$ to optimize data generation following the specialization distribution. In CLSWGAN, $D$ need identify as much as possible real feature from generated feature, while $G$ need trick the discriminator by generated feature that has deviation compared with the real feature. Compared CLSWGAN with GAN, the differences are the addition of the classification loss and the metric method change of the Wasserstein distance loss. According to the inspiration of conditional GAN [14][3], we expect to extend CLSWGAN to the proposed TFGNSCS with a conditional transfer transformation to both $G$ and $D$. In the following we describe the details of TFGNSCS, the novelty of which is that we introduce transfer information into the conditional GAN to generate the more discriminative features for ZSL or GZSL. It is worth noting that the proposed TFGNSCS not only can synthesize the good fidelity features of unseen classes in $S$, but also can refine the performance of the model by the generated features of unseen classes.

We can extend CLSWGAN [1] to the proposed TFGNSCS by transferring the probability model of the generated features from seen to unseen classes for the adversarial training between the generator and the discriminator. The loss has four parts. The first part is constructed based on the improved WGAN[4] and conditional WGAN[1] with the class embedding $c(y_{s})$.

[TABLE]

where $\tilde{x}_{s}=G(z,c(y_{s}))$ is the generative feature of seen classes, $z\in\mathbb{Z}\subset\mathbb{R}^{d_{z}}$ is random Gaussian noise, $c(y_{s})\in\mathbb{C}$ is class embedding of seen classes, $\hat{x}=\alpha x_{s}+(1-\alpha)\tilde{x}$ with $\alpha\thicksim U(0,1)$, and $\lambda$ is the trade-off coefficient. In the loss $L_{WGAN}$, the first two terms compute the Wasserstein distance, and the third term constrains the gradient of $D$ to become unit norm following the straight line between pairs of real and generated point[1].

The second part of the loss is expected to generate CNN feature for adapting a discriminative classifier. In other word, the construction of the classifier can constrain the feature generation of $G$ for balancing their relationship. Therefore, we can maximize the probability of the generated feature $\tilde{x}_{s}$ in the classifier trained by the real feature $x_{s}$, and further minimize the classification loss, which is defined by the negative log likelihood.

[TABLE]

where $\tilde{x}_{s}=G(z,c(y_{s}))$, $y_{s}$ is the class label of $\tilde{x}_{s}$ in seen classes, $P(y_{s}|\tilde{x}_{s};\theta)$ is the probability of $\tilde{x}_{s}$ with the class label $y_{s}$. The probability can be modeled by the $\theta$ parameterizing classification methods, for example the linear softmax classifier or support vector machine. These classification methods can be learned by the real feature and the class label pairs in seen classes.

The first two parts of the loss are the main ideas of CLSWGAN [1]. We propose the novel transfer loss( includes the third part and the forth part of the loss) based on CLSWGAN [1] for considering transfer information. Therefore, we expect that the third part of the loss can process the classification loss of the generated feature $\tilde{x}_{u}$ in unseen classes. However, we can not construct a classifier trained by the real feature $x_{u}$ that is lost in ZSL or GZSL. Because semantic information is complete in the concept of all classes, we can draw support from the relationship between seen and unseen classes in semantic embedding to transfer the classifier model $P(\theta)$. we define that $T(P(\theta))$ is transfer transformation function (It is explained in Section 3.2), and the third part of the loss is the model transfer loss $L_{TRA1}$ that is

[TABLE]

where $\tilde{x}_{u}=G(z,c(y_{u}))$, $c(y_{u})$ is class embedding of semantic description in unseen classes, $Q(y_{u}|\tilde{x}_{u};\theta)$ is the probability of $\tilde{x}_{u}$ with the class label $y_{u}$ based on classifier model $Q(\theta)=T(P(\theta))$. The probability of unseen classes is parameterized by $\theta$. $L_{TRA1}$ is the negative log-likelihood of the transformed model $T(P(\theta))$.In $L_{TRA1}$, we constrain the discrimination of generative features in unseen classes by transferring the classification model of seen classes, and further alleviate the generative feature inconsistence between seen and unseen classes.

The forth part of the loss is constructed based on the discriminator $D$. We expect to capture the identification information of unseen classes in the discriminator $D$. Therefore, the loss $L_{TRA2}$ can be defined as

[TABLE]

where $\tilde{x}_{u}=G(z,c(y_{u}))$,$c(y_{u})$ is class embedding of semantic description in unseen classes. In the loss $L_{TRA2}$, we consider the information of unseen classes for balancing the bias of the discriminator $D$ in seen classes. Therefore, the total loss can be built by the above four parts losses, and full objective can be reformulated as following,

[TABLE]

3.2 Transferring and Classification

In the proposed model, we need obtain three classifier model to generate the discriminative feature for classifying unseen classes.

The first classifier model $P(\theta)$ can be trained by samples of seen classes for describing $L_{CLS}$ to improve the classification performance of the generative feature of seen classes. We can learn model parameter $\theta$ by following equation that is

[TABLE]

where $P(y_{s}|x_{s};\theta)=\frac{\exp(\theta^{T}_{s}x_{s})}{\sum_{s}\exp({\theta^{T}_{s}x_{s}})}$, $P(\theta)=[P(y_{1}|x_{1};\theta),P(y_{2}|x_{2};\theta),...,P(y_{K}|x_{K};\theta)]$, $\theta_{s}$ is a column of $\theta\in\mathbb{R}^{d_{x}\times K}$ that is transformation matrix corresponding to image feature $x$ to $K$ classes probabilities mapping, $\mathbb{T}=S$, and $N_{\mathbb{T}}$ is the number of $\mathbb{T}$.

The second classifier model $Q(\theta)$ can not be learned by unseen classes of samples, which is lost in ZSL and GZSL. Therefore, we construct $Q(\theta)$ by $T(P(\theta))$ for representing $L_{TRA1}$ to enhance the discrimination of the synthesized feature of unseen classes. Given $P(\theta)=A_{s}W_{ss}$, $P(\theta)$ can be decomposed into the sharing part $A_{s}$ and the unique part $W_{ss}$ in probability pattern of seen classes.$Q(\theta)=A_{u}W_{uu}$ means that $Q(\theta)$ can be divided into the sharing part $A_{u}$ and the unique part $W_{uu}$ in probability pattern of unseen classes. $A_{u}=A_{s}W_{su}$ indicates the relationship of the sharing parts between seen and unseen classes. Therefore, transfer transformation function $T(P(\theta))$ can be deduced as

[TABLE]

where$W_{ss}\in\mathbb{R}^{K\times K}$ is the similarity matrix among seen classes,$W_{uu}\in\mathbb{R}^{M\times M}$ is the similarity matrix among unseen classes,$W_{su}\in\mathbb{R}^{K\times M}$ is the similarity matrix between seen and unseen classes based on semantic class prototype graph(In this graph, each class is corresponding to one node, which can be represented by the semantic embedding of each class. The weight between nodes can describe the similarity between classes). These similarity matrices (structure representation) can be measured by cosine distance $d(a,b)$ between the semantic embedding $a$ and $b$ in any two classes.

[TABLE]

[TABLE]

where $W_{ss}(i,j)$ is a element of $W_{ss}$, $i=\{1,2,...,K\}$, $j=\{1,2,...,K\}$, $c(y_{s})_{i}$ or $c(y_{s})_{j}$ denotes the semantic embedding of any class $i$ or $j$ in seen classes,$N_{c(y_{s})_{j}}$ is the neighborhood of $c(y_{s})_{j}$. For selecting the related semantic embedding, the neighborhood number often is set as $5$.

[TABLE]

where $W_{uu}(i,j)$ is a element of $W_{uu}$, $i=\{1,2,...,M\}$, $j=\{1,2,...,M\}$, $c(y_{u})_{i}$ or $c(y_{u})_{j}$ denotes the semantic embedding of any class $i$ or $j$ in unseen classes,$N_{c(y_{u})_{j}}$ is the neighborhood of $c(y_{u})_{j}$.For selecting the related semantic embedding, the neighborhood number often is set as $5$.

[TABLE]

where $W_{su}(i,j)$ is a element of $W_{su}$, $i=\{1,2,...,K\}$, $j=\{1,2,...,M\}$, $c(y_{s})_{i}$ stands for the semantic embedding of any class $i$ in seen classes, while $c(y_{u})_{j}$ is the semantic embedding of any class $j$ in unseen classes,,$N_{c(y_{s})_{j}}$ is the neighborhood of $c(y_{s})_{j}$.For selecting the related semantic embedding, the neighborhood number often is set as $5$.

The third classifier model is be constructed based on the real feature of seen classes $s$ and the synthesized feature of unseen classes $u$ for transforming ZSL to supervised learning.We can learn model parameter $\phi$ by following equation that is

[TABLE]

where $P(y|x;\phi)=\frac{\exp(\phi^{T}_{i}x)}{\sum_{i}\exp({\phi^{T}_{i}x})}$, $\phi_{i}$ is a column of $\phi\in\mathbb{R}^{d_{x}\times N}$ that is regarded as the weight matrix to project the feature $x$ to $N$ categories in a fully connected layer of deep network. In ZSL, $\mathbb{T}=\tilde{U}$, while $\mathbb{T}=S\bigcup\tilde{U}$ in GZSL, $\tilde{U}=\{U,\tilde{x}_{u}\}$.The prediction function $f(x)$ can be defined as

[TABLE]

where in ZSL, $x\in\{x_{u}\}$,$x_{u}$ is the image feature of unseen classes and $y\in\mathbb{Y}_{u}$, while in GZSL, $x\in\{x_{s},x_{u}\}$ and $y\in(\mathbb{Y}_{s}\bigcup\mathbb{Y}_{u})$.

The pseudo code of the TFGNSCS can be shown in Algorithm 1, which has five steps. The first step (line 1) initializes the structure representation of semantic embedding by equation (8), (9), (10)and (11). The second step (line 2) trains the classifier model of seen classes by equation (6) and transfers the classifier model for unseen classes by equation (7).The third step (line 4) updates the discriminator $D$ with equation (5). The forth step (from line 5) updates the generator $G$ with equation (5). The fifth step (from line 7) implements the classifier model training and the label estimating of unseen classes by equation (12) and (13).

4 Experiments

We firstly explain our experimental configuration, and then we demonstrate (a) the comparison results between the proposed method and the state of the arts for ZSL and GZSL on four challenging datasets, (b)our analysis for the base-line methods based on the different loss combination, (c)our extending experiments on the different transfer method and (d)our parameter analysis for image feature generation.

4.1 Datasets

We implement and evaluate the proposed method TFGNSCS for ZSL or GZSL in four challenging datasets, which are Animals with Attributes (AwA)[6], CUB-200-2011 Birds (CUB)[41], SUN Attribute (SUN)[42] and Oxford Flower (FLO)[43]. AwA includes 30475 images, 50 categories and 85 attributes, and belongs to the coarse-grained datasets. CUB, SUN and FLO pertain to the fine-grained datasets. CUB contains 200 birds classes with 312 attributes for a grand total of 11788 images. SUN involves 14340 images from 717 scenes with 102 attributes. FLO has 8189 images from 102 flower classes that can be annotated by the visual description [44]. Table 1 shows the statistics of these datasets.

4.2 Visual and Semantic Feature

ZSL can recognize the visual samples of unseen classes by the completed semantic relation. Visual feature and semantic class embedding should first be extracted or described. Deep network shows the outstanding performance for extracting the discriminative feature from visual or semantic information. Therefore, we use the same description in [1]. We can represent the entire image as the 2048 dimension visual feature from the top layer of the pre-trained 101-layer ResNet[45] based on ImageNet 1K without image pre-processing, network fine-tuned and data augmentation. We can utilize pre-annotated attributes as semantic class embedding, such as AwA with 85 dimension vector, CUB with 312 dimension vector and SUN with 102 dimension. For FLO without the pre-annotated attributes, we extract 1024 dimension feature based on CNN-RNN of fine-grained visual description[44]. In the whole feature extracting process, we obey the ZSL rules that any information of $\mathbb{Y}_{s}$ and $\mathbb{Y}_{u}$ have no crossed set.

4.3 Classification protocols

In ZSL, the test image can be corresponding to an unseen class label in $\mathbb{Y}_{u}$, while this image can be related to any one class label in $\mathbb{Y}_{s}\bigcup\mathbb{Y}_{u}$ in GZSL. For evaluating the performance of ZSL or GZSL, we compute the average per-class top-1 accuracy by dividing the each class accuracy sum by the number of classes, for example, $tr$ stands for average per-class top-1 accuracy on seen classes, $ts$ denotes average per-class top-1 accuracy on unseen classes, and $H=2*tr*ts/(tr+ts)$ represents harmonic mean on $\mathbb{Y}_{s}\bigcup\mathbb{Y}_{u}$ in GZSL. We expect to generate the discriminative feature by the proposed model. For comparing with state of the arts, we preserve the architecture of generator and discriminator in [1]. These architectures both include multi-layer perception (MLP) with leaky rectified linear unit (LReLU), a single layer with 4096 hidden units and a output rectified linear unit (ReLU) layer for learning top max-pooling units of ResNet-101. The noise $z$ come from Gaussian distribution with a unit variance, and the dimensionality of $z$ is same to that of class embedding. We adopt $\lambda=10$ [4], $\beta=0.01$ [1],$\gamma=0.01$ and $\eta=1$ for conveniently comparing the different method result. However, all kinds of hyperparameters often can be obtained by cross-validation based on the specific dataset in fact.

4.4 Comparison with the state-of-the-arts

In this section, because generation adversarial architecture and structure constrains are basic ideas for constructing TFGNSCS, we compare the proposed method with five related state-of-the-arts. The first method is dual-verification network (DVN) constructs and verifies the orthogonal projection between features and attributes with a pairwise manner in the respective spaces[46]. The second method is a hybrid model (HM) includes random attribute selection (RAS) and conditional generative adversarial network (cGAN) for adversarial unseen visual feature synthesis[47]. The third method is visual center learning (VCL) can align the projected semantic center and visual cluster center by minimizing the distance between the synthetic and real center in visual feature space[48]. The forth method is triple verification network (TVN) can construct a unified optimization of regression and compatibility functions for integrating the complementary losses and the mutual regularization [49]. The fifth method is feature generating networks (FGN) can pair a wasserstein GAN with a classification loss to generate sufficiently discriminative CNN features for training softmax classifier in ZSL or GZSL [1]. One thing to note, the above methods are inductive methods, which do not use test datasets for training the learning model, for strictly following ZSL or GZSL setting.

Tab.2 shows the comparison results between the proposed method (TFGNSCS) and five state-of-arts (DVN, HM, VCL, TVN and FGN)for ZSL. The proposed method TFGNSCS has the better result in the various datasets. The performance of TFGNSCS respectively improves $1.7\%$ for AwA, $0.6\%$ for CUB, $2.4\%$ for SUN, and $0.9\%$ for FLO at least.

Tab.3 demonstrates the comparison results between the proposed method (TFGNSCS) and five state-of-arts (DVN, HM, VCL, TVN and FGN)for GZSL. TFGNSCS outperforms five state-of-arts in all datasets. Harmonic mean $H$ can measure the performance of the different methods in the various datasets. The higher value of $H$ indicates the better result for GZSL. $H$ of TFGNSCS respectively improves $4.9$ for AwA, $3.3$ for CUB, $1.2$ for SUN, and $3.1$ for FLO at least.

4.5 Comparison with the base-line methods

The proposed method (TFGNSCS) is constructed based on FGN[1] framework, and extends two loss terms for building transfer feature generating networks. We implement the ablation study for comparing the proposed method with the base-line methods. Therefore, the related base-line methods include FGN[1](the optimization function is $\min_{G}\max_{D}L=\min_{G}\max_{D}L_{WGAN}+\beta L_{CLS}$), TFGNSCS-1(the optimization function is $\min_{G}\max_{D}L=\min_{G}\max_{D}L_{WGAN}+\beta L_{CLS}+\gamma L_{TRA1}$) and TFGNSCS-2(the optimization function is $\min_{G}\max_{D}L=\min_{G}\max_{D}L_{WGAN}+\beta L_{CLS}+\eta L_{TRA2}$). In FGN, the model only considers of the classification loss ($L_{CLS}$) of seen classes based on the generative adversarial framework. In TFGNSCS-1, the model integrates the classification loss ($L_{TRA1}$) of unseen classes with FGN model based on semantic structure transfer. In TFGNSCS-2, the model combines the discriminator loss ($L_{TRA1}$) for unseen classes with FGN model. TFGNSCS model considers all of these factors for transferring and balancing the information between seen and unseen classes.

Tab.4 shows the experimental results of the proposed method TFGNSCS and the base-line methods for ZSL. The performance TFGNSCS outperforms that of the other methods. The improvement of TFGNSCS is $2.3\%$ for AwA, $2.9\%$ for CUB, $4\%$ for SUN and $0.9\%$ for FLO at least. Tab.5 demonstrates that the performance TFGNSCS outperforms that of the base-line methods for GZSL. $H$ of TFGNSCS respectively improves $1.0$ for AwA, $1.2$ for CUB, $0.1$ for SUN, and $1.0$ for FLO** at least. For ZSL and GZSL, the performance of TFGNSCS-1 is better than that of TFGNSCS-2 and FGN, while the performance of FGN is worse than that TFGNSCS-1 and TFGNSCS-2. $L_{TRA1}$ in TFGNSCS-1 is individually considered to ascend the performance of the model, and $L_{TRA2}$ in TFGNSCS-2 is individually considered also to improve the performance of the model. However, $L_{TRA2}$ individually enhance the difference of the discriminator between seen and unseen classes, whereas transfer factor $L_{TRA1}$ can weaken these imbalance information between seen and unseen classes. Therefore, the combination of $L_{TRA1}$ and $L_{TRA2}$ in TFGNSCS can make $L_{TRA2}$ boost the transfer characteristic of $L_{TRA1}$ for improving the performance of ZSL and GZSL.

4.6 Comparison with the different transfer methods

Transfer method is an key point for constructing generating network model. In this paper, we focus on the transfer method from the classifier model $P(\theta)$ of seen classes to the classifier model $Q(\theta)$ of unseen classes. The equation (7) shows the transformation relationship between $P(\theta)$ and $Q(\theta)$. Beside this way, while we incorporate image feature of unseen classes into a semantic class prototype graph, ZSL can be regard as an extended absorbing Markov chain process on this graph [22]. Therefore, a alternative transfer transformation function $T(P(\theta))$ is

[TABLE]

where $I\in\mathbb{R}^{K\times K}$. We use the transfer method to learn model, which can be expressed as TFGNSCS-alt. In Tab.6 and Tab.7, we find that difference between transfer methods is slight for ZSL or GZSL. The main reason is that the different transfer methods can both adjust the imbalance information between seen and unseen classes for improving the discrimination of generative feature by adversarial learning.

4.7 Parameter analysis

In TFGNSCS, the number of generative features directly impacts on the performance of ZSL or GZSL. Therefore we select the number of generative features from $1,2,6,10,30,50,100,200,300$ to construct the different classification model for comparing TFGNSCS with the base-line methods. Figure 5 shows TFGNSCS outperforms the base-line methods. Especialy there is the significant boost of classification accuracy with number increasing from 1 to 50 of generative features for unseen classes, e.g. $42.1$ to $67.7$ on FLO and $33.3$ to $57.0$ on CUB in TFGNSCS. Figure 6 also demonstrates the significant boost of harmonic mean with number increasing from 1 to 50 of generative features for unseen classes, e.g. $48.1$ to $67.7$ on FLO and $32.1$ to $48.6$ on CUB in TFGNSCS. It shows that TFGNSCS has the better adaptability than other methods.

4.8 Experimental results analysis

In experiments, we compare the proposed method with eight methods, which include five kinds of state-of-the-art methods (DVN [46],HM[47],VCL[48],TVN[49] and FGN[1] in section 4.4), three kinds of base-line methods(FGN[1],TFGNSCS-1 and TFGNSCS-2 in section 4.5) and a alternative transfer method (TFGNSCS-alt in section 4.6). These methods can construct the related model to bridge the gaps between visual and semantic information for ZSL or GZSL. In contrast to these methods, the proposed method focuses on mining the transfer information in generation adversarial framework for the discriminative synthetic feature. From these experiment, we have the following observations.

The performance of TFGNSCS is better than that of five kinds of state-of-the-art methods (DVN,HM,VCL,TVN and FGN in section 4.4) for ZSL, and the performance of TFGNSCS outperforms that of these methods for GZSL. This situation of the main reason is that TFGNSCS try to balance the difference between seen and unseen classes by transfer losses. Therefore, in ZSL setting, the classification accuracy of unseen classes is higher than other methods on four datasets, moreover, the betterment is noticeable for harmonic mean in GZSL setting. Especially, harmonic mean of FGN and TFGNSCS significantly exceeds that of other state-of-the-art methods for GZSL in the different datasets.

• 2.

The performance improvement of TFGNSCS is different in three base-line approaches (FGN,TFGNSCS-1,TFGNSCS-2) for ZSL or GZSL. The advance of TFGNSCS can be found for ZSL in four datasets, while the better improvement can be demonstrated for GZSL in all datasets. In there,the outstanding betterment is harmonic mean of GZSL in AwA and FLO. It shows that transfer method can enhance the discrimination of the generative features by transfer losses in adversarial networks, and further validates that semantic structure transfer is effective for constructing the learning model in ZSL or GZSL.

• 3.

The different transfer methods have the similar performance for ZSL and GZSL in all datasets. In this paper, two kinds of transfer method both is built based on the semantic classes of graph, which can represent the distribution structure of classes. The difference of these transfer methods is on the different way of the structure propagation. The structure propagation of that equation (7) in TFGNSCS is formed by drawing support from the relationship of the sharing parts in respective classification model on the seen or unseen classes, while the structure propagation of that equation (14) in TFGNSCS-alt is constructed based on an extended absorbing Markov chain process. In the learning process of the adversarial networks, this difference of the structure propagation is trivial for ZSL and GZSL.

• 4.

The two loss parts (the unseen classes classification loss $L_{TRA1}$ and the generated features discrimination loss $L_{TRA2}$ ) of transfer loss have the diverse effect to improve the performance of ZSL and GZSL. The unseen classes classification loss $L_{TRA1}$ can boost the performance of ZSL and GZSL, while the generated features discrimination loss $L_{TRA2}$ play a fewer role for this melioration. However, the integration of these loss can further ameliorate the performance of ZSL and GZSL. The loss $L_{TRA2}$ is a assistant method for ascending the performance of TFGNSCS, and it’s role depends on the quality of the generative feature by $L_{TRA1}$.

• 5.

The number of the generative features influences the performance of ZSL and GZSL. The increasing number of the generative features makes the performance improve for ZSL, and this situation is more obvious for GZSL. It shows that the proposed method TFGNSCS can synthesize the more discriminative feature at the same number because of the transfer loss contribution for model construction.

5 Conclusion

We have proposed transfer feature generating networks with semantic classes structure (TFGNSCS) method to address imbalance between seen and unseen classes in ZSL and GZSL. TFGNSCS can not only adapt the semantic structure relationship between seen and unseen classes to a uniform generative features framework, but also model the difference of generating features by balancing transfer information between seen and unseen classes in networks. Furthermore, TFGNSCS can combine a Wasserstein generative adversarial network with classification loss and transfer loss to generate enough CNN feature for improving ZSL and GZSL. At last, the optimization learning of the TFGNSCS can obtain both the transfer feature generating networks and the more discriminative features. For evaluating the proposed TFGNSCS, we carry out the comparison experiments about the state of the art methods, the baseline methods, the other transfer method and parameter analysis on AwA, CUB, SUN and FLO datasets. Experiment results demonstrate the TFGNSCS gets the promising results in ZSL and GZSL.

6 Acknowledgements

The authors would like to thank the anonymous reviewers for their insightful comments that help improve the quality of this paper. Especially, The authors thank to Dr. Yongqin Xian from MPI for Informatics, who provided the data source to us. This work was supported by NSFC (Program No.61771386,Program No.61671376 and Program No.61671374), Natural Science Basic Research Plan in Shaanxi Province of China (Program No.2017JZ020).