Cross-modal Subspace Learning via Kernel Correlation Maximization and Discriminative Structure Preserving

TL;DR

This paper introduces a novel cross-modal subspace learning framework that maximizes kernel correlation and preserves semantic structure, improving the alignment of heterogeneous data modalities.

Contribution

It proposes a new framework combining kernel correlation maximization with semantic structure preservation using a shared semantic graph and HSIC.

Findings

Outperforms classic subspace learning methods on three datasets

Effectively preserves semantic neighbor relationships within modalities

Ensures inter-modality correlation and intra-modality structure are maintained

Abstract

The measure between heterogeneous data is still an open problem. Many research works have been developed to learn a common subspace where the similarity between different modalities can be calculated directly. However, most of existing works focus on learning a latent subspace but the semantically structural information is not well preserved. Thus, these approaches cannot get desired results. In this paper, we propose a novel framework, termed Cross-modal subspace learning via Kernel correlation maximization and Discriminative structure-preserving (CKD), to solve this problem in two aspects. Firstly, we construct a shared semantic graph to make each modality data preserve the neighbor relationship semantically. Secondly, we introduce the Hilbert-Schmidt Independence Criteria (HSIC) to ensure the consistency between feature-similarity and semantic-similarity of samples. Our model not only considers the inter-modality correlation by maximizing the kernel correlation but also preserves the semantically structural information within each modality. The extensive experiments are performed to evaluate the proposed framework on the three public datasets. The experimental results demonstrated that the proposed CKD is competitive compared with the classic subspace learning methods.

Figures

Tables

Table 1. Table 1: Statistics of three standard datasets

DataSets	Pascal-Sentence	MIRFlickr	NUS-WIDE
Data Set Size	1000	16738	190420
Training Set Size	600	5000	5212
Retrieval Set Size	600	5000	5212
Query Set Size	400	836	3475
Num. of Labels	20	24	21

Table 2. Table 2: The MAP results on Pascal-Sentence, MIRFlickr and NUS-WIDE

Datasets	Approaches	I2T	T2I	Avg.
Pascal-Sentence	CCA	0.0501	0.0456	0.0479
	KCCA	0.0376	0.0402	0.0389
	ml-CCA	0.0422	0.0329	0.0376
	CKD($\beta$=0)	0.0736	0.1300	0.1018
	KDM	0.1729	0.1992	0.1861
	CKD	0.2143	0.2806	0.2475
MIRFlickr	CCA	0.5466	0.5477	0.5472
	KCCA	0.5521	0.5529	0.5525
	ml-CCA	0.5309	0.5302	0.5306
	CKD($\beta$=0)	0.5602	0.5595	0.5599
	KDM	0.5951	0.5823	0.5887
	CKD	0.6103	0.5933	0.6018
NUS-WIDE	CCA	0.3099	0.3103	0.3101
	KCCA	0.3088	0.3174	0.3096
	ml-CCA	0.2787	0.2801	0.2794
	CKD($\beta$=0)	0.3170	0.3164	0.3167
	KDM	0.3247	0.3118	0.3183
	CKD	0.4149	0.4211	0.4180

Table 3. Table 3: The comparative MAP results on NUS-WIDE

Methods	I2T	T2I	Avg.
Multimodal DBN	0.2013	0.2594	0.2303
Bimodal-AE	0.3271	0.3693	0.3482
Corr-AE	0.3658	0.4172	0.3915
DCCA	0.4844	0.5088	0.4966
CMDN	0.4923	0.5151	0.5037
ACMR	0.5437	0.5376	0.5407
Deep-SM	0.6793	0.6930	0.6862
DCKD(Proposed)	0.6827	0.7026	0.6927

Table 4. Table 4: The comparison of training and testing time(seconds) on Pascal-Sentence dataset

Time $\backslash$ Methods	CCA	KCCA	ml-CCA	KDM	CKD
Training time	1.3462	21.2693	3410.0332	419.3401	410.7084
Testing time	0.0496	0.0903	0.2269	0.04699	0.04576

Equations

$$HSIC=(n-1{)}^{-2}tr(K_{x}H{K}_{z}H)$$

$$\begin{array}{rl}\underset{P_1,P_2}{\max }& tr(H{K}_{X_1}H{K}_{X_2})+tr(H{K}_{X_1}H{K}_Y)+tr(H{K}_{X_2}H{K}_Y)\\ & s.t.P_1^T{P}_1=I,P_2^T{P}_2=I\end{array}$$

$$S_{ij}=\frac{y_i\cdot y_j}{\Vert y_i{\Vert }_2\Vert y_j{\Vert }_2}$$

$$\begin{array}{rl}\underset{P_1,P_2}{\min }& {\alpha }_{1}tr(P_1^T{X}_1^{T}L{X}_1{P}_1)+{\alpha }_{2}tr(P_2^T{X}_2^{T}L{X}_2{P}_2)\\ & s.t.P_1^T{P}_1=I,P_2^T{P}_2=I\end{array}$$

$$\begin{array}{rl}\underset{P_1,P_2}{\min }& {\alpha }_1(tr(P_1^T{X}_1^{T}L{X}_1{P}_1)+{\lambda }_1\Vert P_1{\Vert }_{2,1})\\ & +{\alpha }_2(tr(P_2^T{X}_2^{T}L{X}_2{P}_2)+{\lambda }_2\Vert P_2{\Vert }_{2,1})\\ & s.t.P_1^T{P}_1=I,P_2^T{P}_2=I\end{array}$$

$$\begin{array}{rl}\underset{P_1,P_2}{\min }& {\alpha }_1(tr(P_1^T{X}_1^{T}L{X}_1{P}_1)+{\lambda }_1\Vert P_1{\Vert }_{2,1})\\ & +{\alpha }_2(tr(P_2^T{X}_2^{T}L{X}_2{P}_2)+{\lambda }_2\Vert P_2{\Vert }_{2,1})\\ & +\beta (-tr(H{K}_{X_1}H{K}_{X_2})-tr(H{K}_{X_1}H{K}_Y)\\ & -tr(H{K}_{X_2}H{K}_Y))\\ & s.t.P_1^T{P}_1=I,P_2^T{P}_2=I\end{array}$$

$$\begin{array}{rl}\underset{P_1,P_2}{\min }& -tr(H{K}_{X_1}H{K}_{X_2})-tr(H{K}_{X_1}H{K}_Y)\\ & -tr(H{K}_{X_2}H{K}_Y)\\ & +{\alpha }_1(tr(P_1^T{X}_1^{T}L{X}_1{P}_1)+{\lambda }_{1}tr(P_1^T{D}_1{P}_1))\\ & +{\alpha }_2(tr(P_2^T{X}_2^{T}L{X}_2{P}_2)+{\lambda }_{2}tr(P_2^T{D}_2{P}_2))\\ & s.t.P_1^T{P}_1=I,P_2^T{P}_2=I\end{array}$$

$$\begin{array}{rl}\underset{P_1}{\max }& tr(P_1^T{Q}_1{P}_1)\\ & s.t.P_1^T{P}_1=I\end{array}$$

$$\begin{array}{rl}\underset{P_2}{\max }& tr(P_2^T{Q}_2{P}_2)\\ & s.t.P_2^T{P}_2=I\end{array}$$

$$\frac{\Vert f{\Vert }_2}{2\Vert g{\Vert }_2}-\Vert f{\Vert }_2\ge \frac{\Vert g{\Vert }_2}{2\Vert g{\Vert }_2}-\Vert g{\Vert }_2$$

$$\begin{array}{rl}\underset{P_1}{\min }& -tr(H{K}_{X_1}H{K}_{X_2})-tr(H{K}_{X_1}H{K}_Y)\\ & +{\alpha }_1(tr(P_1^T{X}_1^{T}L{X}_1{P}_1)+{\lambda }_{1}tr(P_1^T{D}_1{P}_1))\\ & s.t.P_1^T{P}_1=I\end{array}$$

$$\begin{array}{rl}{P}_1^{(k+1)}& ={\mathrm{argmin}}_{P_1}({\lambda }_{1}tr(P_1^T{D}_1{P}_1)-W(P_1))\\ & \Rightarrow {\lambda }_{1}tr(P_1^{(k+1{)}^T}{D}_1^{(k)}{P}_1^{(k+1)})-W(P_1^{(k+1)})\\ & \le {\lambda }_{1}tr(P_1^{(k{)}^T}{D}_1^{(k)}{P}_1^{(k)})-W(P_1^{(k)})\\ & \Rightarrow {\lambda }_1\sum _i^{d_1}\frac{\Vert P_1^{\cdot \cdot i(k+1)}{\Vert }_2^2}{2\Vert P_1^{\cdot \cdot i(k)}{\Vert }_2^2}-W(P_1^{(k+1)})\\ & \le {\lambda }_1\sum _i^{d_1}\frac{\Vert P_1^{\cdot \cdot i(k)}{\Vert }_2^2}{2\Vert P_1^{\cdot \cdot i(k)}{\Vert }_2^2}-W(P_1^{(k)})\\ & \Rightarrow {\lambda }_1(\sum _i^{d_1}\frac{\Vert P_1^{\cdot \cdot i(k+1)}{\Vert }_2^2}{2\Vert P_1^{\cdot \cdot i(k)}{\Vert }_2^2}-\Vert P_1^{\cdot \cdot i(k+1)}{\Vert }_{2,1})\\ & +{\lambda }_1\Vert P_1^{\cdot \cdot i(k+1)}{\Vert }_{2,1}-W(P_1^{(k+1)})\\ & \le {\lambda }_1(\sum _i^{d_1}\frac{\Vert P_1^{\cdot \cdot i(k)}{\Vert }_2^2}{2\Vert P_1^{\cdot \cdot i(k)}{\Vert }_2^2}-\Vert P_1^{\cdot \cdot i(k)}{\Vert }_{2,1})\\ & +{\lambda }_1\Vert P_1^{\cdot \cdot i(k)}{\Vert }_{2,1}-W(P_1^{(k)}).\\ & \end{array}$$

$${\lambda }_1\Vert P_1^{\cdot \cdot i(k+1)}{\Vert }_{2,1}-W(P_1^{(k+1)})\le {\lambda }_1\Vert P_1^{\cdot \cdot i(k)}{\Vert }_{2,1}-W(P_1^{(k)})$$

$$\Rightarrow T(P_1^{(k+1)})\le T(P_1^{(k)})$$

$$AP(q)=\frac{1}{l_q}\sum _{m=1}^R{P}_q(m){\delta }_q(m)$$

Full text

1 Introduction

Recently, the fast development of the Internet and the explosive growth of multimedia including text, image, video, audio has greatly enriched people’s life but magnified the challenge of information retrieval. Representative image retrieval methods, such as Region- based image retrieval [1], Color-based image retrieval [2], Contour Points Distribution Histogram(CPDH) [3], Inverse Document Frequency (IDF) [4], Content-based image retrieval [5], are not directly applied in multimodal. Multimodal data refers to those data of different types but with the same content information, for example, recording video clips, music, photos and tweets of a concert. Cross-modal retrieval which aims to take one type of data as the query to return the relevant data of another type has attracted much attention.

   The cross-modal retrieval methods need to solve a basic problem, i.e. how to measure the relevance between heterogeneous modalities. There are two strategies to solve this challenge. One is to directly calculate the similarity based on the known relationship between cross-modal data [6, 7, 8]. The other is to learn a latent common subspace where the distance between different modalities data can be measured, which is also termed as cross-modal subspace learning. The subspace learning models include unsupervised approaches [9, 10, 11], supervised approaches [13, 14, 12, 15, 16, 17]. Unsupervised methods usually use the intrinsical characteristic of data and the correlation between multimodal data to learn common subspace representation. There exist some inherent correlation among multiple modalities, since pair-wise multiple modalities describe the same semantic object. Canonical Correlation Analysis (CCA), kullback Leibler (KL) divergence, Hilbert-Schmidt Independence Criterion (HSIC) are widely used to measure the correlation between multiple modalities. However, unsupervised methods encounter bottlenecks and do not obtain satisfactory results bacause the available discriminative information is inadequate. To solve the problem, supervised methods introduce the semantic label information to learn the discriminative feature representation of each modality. Although existing supervised methods have achieved reasonable performance, they are still some drawbacks. Most of these methods do not explore the semantically structural information and the correlation among multiple modalities simultaneously in the process of learning a latent subspace. Actually, the semantically structural information is very important to make the subspace representation more discriminative.

   In this paper, we propose a novel framework which preserves both the shared semantic structure and the correlation among multiple modalites. The supervised information, semantic structure and multimodal data are collaboratively incorporated in a unified framework to uncover a common subspace for cross-modal retrieval, as illustrated in Fig. 1. The main contributions of our work are summarized as follows:

   (1) The proposed learning model combines subspace learning with feature selection and semantic structure-preserving into a joint framework. A shared semantic graph is constructed to learn the discriminative feature representation of each modality. In addition, the convergence of the algorithm is analyzed theoretically.

   (2) Extensive experiments are performed on three widely-used datasets and the experimental results show the superiority and effectiveness of our model.

   Structurally, the rest of this paper falls into four parts. In Section 2, we review related works to cross-modal retrieval. In Section 3, we present the proposed CKD model and joint optimization process. The experimental results and analysis are provided in Section 4. In Section 5, we draw the conclusions of the paper.

2 Related work

Cross-modal retrieval has been a popular topic in multimedia applications. Many methods are developed to realize the retrieval tasks in the past decades. In this section, we preliminarily review the related works of this field. According to the different types of targeted representation, cross-modal retrieval methods are divided into two branches: real-valued and hash code representation learning. The hash code representation learning is referred to as cross-modal hashing. Cross-modal hashing approaches [19, 20, 21, 22, 23, 24] aim to learn a common Hamming space. Since cross-modal hashing methods are more geared towards retrieval efficiency, their focus is not on retrieval effectiveness. Real-valued representation learning approaches are designed to learn a common subspace where the similarity between different modalities can be measured directly.These approaches are roughly grouped into two paradigms: unsupervised subspace learning and supervised ones.

Unsupervised subspace learning methods explore the structure, distribution and topological information of data to learn common subspace representation for each modality. Canonical Correlation Analysis (CCA) [25] is the most representative unsupervised method. However, unsupervised methods do not obtain satisfying retrieval precision because the encoded features lack discriminative ability. Supervised methods exploit the semantic label information to learn a more discriminative common subspace. An example is three-view CCA (CCA-3V) [26] which regards the semantic label as a feature view to learn a common space for cross-modal retrieval. Multi-label CCA (ml-CCA) [45] is proposed to perform cross-modal retrieval by utilizing multi-label annotations. Semantic Correlation Matching (SCM) [27] uses the logistic regression to learn semantic subspace representation for each modality. Joint Feature Selection and Subspace Learning (JFSSL) [28] combines common subspace learning and coupled feature selection into a unified framework.

In recent years, many deep learning models have achieved promising performance. Deep CCA (DCCA) [29] extends the linear CCA to nonlinear one. Adversarial Cross-Modal Retrieval (ACMR) [30] is proposed to seek an effective common subspace based on adversarial learning. Wei et al proposed deep-SM [31] to learn a common semantic space where the probabilistic scores produced by a deep neural network are viewed as the semantic features. Deep Supervised Cross-Modal Retrieval (DSCMR) [32] aims to find a common representation space by minimizing the discrimination loss and modality invariance loss simultaneously. Adversarial Cross-Modal Embedding (ACME) [33] is proposed to resolve the cross-modal retrieval task in food domains.

The approach proposed in this paper employs the label information to learn effective and discriminative subspace representation. Since multimodal data appears in pairs and is semantically relevant, it is necessary to explore the correlation among multiple modalities. Besides, learned subspace representation should preserve the semantic structure, i.e. the closer the semantic relationship among samples the nearer the distance in common subspace and vice versa. Unlike most of the existing subspace learning methods, our model maximizes the correlation among multi-modal data and preserves the semantic structure within each modality simultaneously.

3 Our Method

3.1 Problem Formulation

Assume that $M$ modalities denotes as $X=\{X^{(1)},X^{(2)},...,X^{(M)}\}$. The $v$-th modality $X^{(v)}=\{X^{(v)}_{1},X^{(v)}_{2},...,X^{(v)}_{n}\}\in R^{n\times d_{v}}$($v=1,...,M$) contains $n$ samples with $d_{v}$ dimension. The label matrix is denoted by $Y=[y_{1}^{T},y_{2}^{T},...y_{n}^{T}]^{T}\in R^{n\times c}$, where $c$ is the number of categories. $y_{ik}=1$ if the $i$-th sample belongs to the $k$-th class; $y_{ik}=0$ otherwise. Without loss of generality, the samples of each modality are zero-centered, i.e. $\sum^{n}_{i=1}X^{(v)}_{i}=0$, ($v=1,2...M$). The aims of this paper is to learn isomorphic feature representation for heterogeneous multi-modal data.

3.2 Hilbert-Schmidt independence criteria (HSIC)

Given two mapping functions with $\phi(x):x\rightarrow R^{d}$ and $\phi(z):z\rightarrow R^{d}$. The associated positive definite kernel $K_{x}$ and $K_{z}$ are formulated as $K_{x}=<\phi(x),\phi(x)>$ and $K_{z}=<\phi(z),\phi(z)>$ respectively. The cross-covariance function between $x$ and $z$ is denoted as $C_{xz}=E_{xz}[(\phi(x)-u_{x})\otimes(\phi(z)-u_{z})]$, where $u_{x}$ and $u_{z}$ are the expectation of $\phi(x)$ and $\phi(z)$ respectively. The Hilbert-Schmidt norm of $C_{xz}$ is defined as $HSIC=\left\|C_{xz}\right\|_{HS}^{2}$. For $n$ paired data samples $D=\{(x_{1},z_{1}),...,(x_{n},z_{n})\}$, An empirical expression of Hilbert-Schmidt independence criteria (HSIC) [34, 35, 36] is defined as

[TABLE]

where $H=\bm{I}-\frac{1}{n}\bm{1}_{n}\bm{1}_{n}^{T}$ is a centering matrix. The larger the $HSIC$, the stronger the correlation between $x$ and $z$.

3.3 Model

For simplicity, we discuss our algorithm based on two modalities, i.e. Image and Text. It is easy to extend to the case with more modalities.

3.3.1 Kernel Correlation Maximization

Our model adopts the HSIC to maximize the kernel dependence among multi-modal data. Multi-modal data are projected into the Hilbert space where we can calculate the kernel matrix $K_{X_{v}}=<V_{v},V_{v}>=V_{v}V_{v}^{T}$, where $V_{v}=\phi_{v}(X^{(v)})=X^{(v)}P_{v}$ ($v=1,2$) and $P_{v}$ is the projection matrix of the $v$-th modal data. As the kernel matrix itself represents, in essence, the similarity relationship among samples, our model preserves the similarity relationship among samples for each modality, which is called the intra-modality correlation. Semantic label shared by multiple modalities is regarded as a semantic modality. Likewise, the kernel matrix of the semantic modality is signified as $K_{Y}=<Y,Y>=YY^{T}$. Our model preserves the consistence between feature-similarity and semantic-similarity of samples via maximizing kernel dependence. According to the definition (1), the objective formulation can be given as follows:

[TABLE]

where $K_{X_{1}}=X_{1}P_{1}P_{1}^{T}X_{1}^{T},K_{X_{2}}=X_{2}P_{2}P_{2}^{T}X_{2}^{T},K_{Y}=YY^{T}$. In this section, the inter-modality correlation and intra-modality similarity relationship are taken into account simultaneously in our model.

3.3.2 Discriminative Structure-Preserving

Although different modalities locate isomeric spaces, they share the same semantic information. We calculate the cosine similarity among samples by employing their semantic label vectors. Specifically, the similarity between the $i$-th and the $j$-th sample is defined as follows

[TABLE]

where $\rVert y_{i}\rVert_{2}$ denote the $L_{2}$-norm of the vector $y_{i}$. We hope that the semantically structural relationship among samples is preserved in Hilbert space. That is to say, the closer the semantic relationship among samples the nearer the distance in the common Hilbert space. The problem can be formulated as the following objective function (4).

[TABLE]

where $L=diag(S\bm{1})-S$ denotes a graph Laplacian matrix and $\alpha_{1}$ and $\alpha_{2}$ are two adjustable parameters. As discussed in some literature [37] [38], the $l_{2,1}$-norm constraint has some advantages of feature selection, sparsity, and robustness to noise. In our model, we impose the $l_{2,1}$-norm constraint on projection matrices to learn more discriminative features and remove the redundant features. The objective function (4) can be rewritten as follows

[TABLE]

where $\lambda_{1}$ and $\lambda_{2}$ are two trade-off parameters.

We integrate kernel dependence maximization and discriminative structure-preserving into a joint framework by combining (2) and (5). The overall objective function can be written as follows

[TABLE]

where $\beta$ is an adjustable parameter. We usually set $\beta=1$ for simplicity. The case with $\beta=0$ implies that our model only considers the discrimiantive structure-preserving.

3.4 Optimization

To optimize variables conveniently, we transform the $l_{2,1}$-norm constraint term into $tr(P_{v}^{T}D_{v}P_{v})$ by adding an intermediate variable $D_{v}=diag(\frac{1}{\rVert P_{v}^{\cdot\cdot i}\rVert_{2}})$, where $P_{v}^{\cdot\cdot i}$ is the $i$-th row of $P_{v}(v=1,2)$. The objective function (6) can be rewritten as

[TABLE]

where $K_{X_{1}}=X_{1}P_{1}P_{1}^{T}X_{1}^{T},K_{X_{2}}=X_{2}P_{2}P_{2}^{T}X_{2}^{T},K_{Y}=YY^{T}$.

3.4.1 Optimization of $P_{1}$

Keeping only the terms relating to $P_{1}$, we can obtain

[TABLE]

where $Q_{1}=X_{1}^{T}HX_{2}P_{2}P_{2}^{T}X_{2}^{T}HX_{1}+X_{1}^{T}HYY^{T}HX_{1}-\alpha_{1}X_{1}^{T}LX_{1}-\alpha_{1}\lambda_{1}D_{1}$. The optimal $P_{1}$ in (8) can be obtained via the eigenvalue decomposition on $Q_{1}$.

3.4.2 Optimization of $P_{2}$

Keeping only the terms relating to $P_{2}$, we can obtain

[TABLE]

where $Q_{2}=X_{2}^{T}HX_{1}P_{1}P_{1}^{T}X_{1}^{T}HX_{2}+X_{2}^{T}HYY^{T}HX_{2}-\alpha_{2}X_{2}^{T}LX_{2}-\alpha_{2}\lambda_{2}D_{2}$. Being similar to $P_{1}$, the optimal solution of $P_{2}$ can be got by the eigenvalue decomposition on $Q_{2}$.

3.5 Convergence analysis

The detail optimization procedure is summarized in Algorithm 1. The process is repeated until the algorithm converges. The convergence curves on NUS-WIDE, Pascal-Sentence and, MIRFlickr25k are plotted in Fig.2, which indicates that our method converges quickly.

$\bm{Theorem1:}$ The objective function (7) based on the optimizing rule (Algorithm 1) is decreasing monotonically, and it converges to the global minimum value.

$\bm{Lemma1:}$ Given any nonzero $f\in R^{m}$ and $g\in R^{m}$, then the following formula (10) holds. Please refer to [39] for details.

[TABLE]

In order to prove the $\bm{Theorem1}$, we introduce the $\bm{Lemma1}$. The detail proof about $\bm{Theorem1}$ is given as follows.

$\bm{Proof:}$ The optimization of $P_{1}$ and $P_{2}$ are symmetrical in Algorithm 1, thus we just consider to prove one of them. The detailed proof with respect to the optimization of $P_{1}$ is provided below. The optimization problem about $P_{1}$ is written as follows

[TABLE]

Letting $W(P_{1})=tr(HK_{X_{1}}HK_{X_{2}})+tr(HK_{X_{1}}HK_{Y})-\alpha_{1}tr(P_{1}^{T}X_{1}^{T}LX_{1}P_{1})$, then the object function in Eq.(11) becomes $T(P_{1})=\lambda_{1}tr(P_{1}^{T}D_{1}P_{1})-W(P_{1})$. For the $k$-th iteration,

[TABLE]

By virtue of $\bm{Lemma1}$, we can obtain

[TABLE]

then

[TABLE]

Similarly, we can also prove that $T(P_{2}^{(k+1)})\leq T(P_{2}^{(k)})$ for the optimization of $P_{2}$. So, we have $T(P_{1}^{(k+1)},P_{2}^{(k+1)})\leq T(P_{1}^{(k)},P_{2}^{(k)})$. The proposed model based on the updating rule (Algorithm 1) is decreasing monotonically. Since the optimization problem is convex, the objective function finally converges to the global optimal solution.

4 Experiments

In this section, we conduct some experiments on three benchmark datasets, i.e. Pascal-Sentence [40], MIRFlickr [41] and NUS-WIDE [42]. Two widely-used evaluation protocols are introduced to evaluate our algorithm.

4.1 Datasets

Pascal-Sentence consists of 1000 samples with image-text pairs. Each image is described in several sentences. The dataset is divided into 20 categories and each category includes 50 samples. We randomly select 30 samples from each category as the training set and the rest for the testing set. For each image, we employ a convolutional neural network to extract 4096-dimension CNN features. For text features, We utilize the LDA model to get the probability of each topic, and the 100-dimensional probability vector is used to represent text features.

   MIRFlickr contains original 25,000 images crawled from the Flickr website. Each image and its associated textual tags is called an instance. Each instance is manually classified into some of the 24 classes. We only keep those instances whose textual tags appear at least 20 times and remove those instances without annotated labels or any textual tags. Each image is represented by a 150-dimensional edge histogram vector, and each text is represented as a 500-dimensional vector derived from PCA on the bag-of-words vector. We randomly select 5% of the instances as the query set and 30% of the instance as the training set.

   NUS-WIDE is a subset sampled from a real-world web image dataset including 190,420 image-text pairs with 21 possible labels. For each pair, image is represented by 500-dimensional SIFT BoVW features and 1000-dimensional text annotations for text. The dataset contains 8,687 image-text pairs which are divided into two parts: 5,212 pairs for training and 3,475 pairs for testing.

4.2 Experimental setting

The proposed CKD in this paper is a supervised, kernel-based and correlation-based method. We compare our algorithm with the following methods: CCA [43] ( Correlation-based method); KCCA [44] (Kernel-based and Correlation-based method); ml-CCA [45] (Supervised and Correlation-based method); KDM [34] (Supervised, Kernel-based and Correlation-based method). Besides, we compare our algorithm with the case where $\beta$ is set as zero. We tune the parameter $d$ in the range of {10,20,30,40,50,60}. $\lambda_{1}$ and $\lambda_{2}$ are two coefficients of the regularization term and they are fixed as 0.01 in experiments. $\alpha_{1}$ and $\alpha_{2}$ are trade-off parameters in the objective function. We set the possible values of $\alpha_{1}$ and $\alpha_{2}$ in the range of {1e-5,1e-4,1e-3,1e-2,1e-1,1,10,1e2,1e3,1e4,1e5 } empirically. The best results are reported in this paper. Our experiments are implemented on MATLAB 2016b and Windows 10 (64-Bit) platform based on desktop machine with 12 GB memory and 4-core 3.6GHz CPU, and the model of the CPU is Intel(R) CORE(TM) i7-7700.

4.3 Evaluation Protocol

There are many evaluation metrics in the information retrieval area. We introduce two commonly used indicators, i.e. Cumulative Match Characteristic Curve (CMC) and Mean Average Precision (MAP). For a query $q$, the Average Precision (AP) is defined as E.q.(15)

[TABLE]

where $\delta_{q}(m)=1$ if the result of position $m$ is right and $\delta_{q}(m)=0$ otherwise; $l_{q}$ represents the correct statistics of top $R$ retrieval results; The average value of AP of all queries is called MAP which indicates a better performance with larger value.

   CMC is the probability statistics that the true retrieval results appear in different-sized candidate lists. Specifically, if retrieval results contain one or more objects classified into the same class with query data, we think that this query can match true object. Assuming that the length of retrieval results is fixed as $m$, the rate of true match in all query is denoted as CMCrank-m.

4.4 Results

We perform some experiments for two typical retrieval tasks: Image query Text database and Text query Image database which are abbreviated as ’I2T’ and ’T2I’ respectively in this paper. I2T indicates known an image as a query, to retrieve the same semantic texts wit the image from the text database. By analogy, the meaning of T2I is as follows: Given a text, to search for images being similar content to the text from the image database. In this paper, the normalized correlation (NC) [27] which is an effective measurement strategy is adopted to measure the similarity among data. Table 1 shows the MAP results on Pascal-Sentence, MIRFlickr, and NUS-WIDE. As reported in Table 1, we can observe that CKD outperforms the compared methods. Specifically, CKD achieves average improvements of 6.14%, 1.31%, and 9.97% over the best baselines on Pascal-Sentence, MIRFlickr, and NUS-WIDE respectively. Besides, we set the candidate list size in rank {5,10,15,20,25,30}. according to the CMC protocol, we conduct some experiments to further validate the effectiveness of CKD. Fig. 3 and Fig. 4 show the performance variation of all approaches with respect to different-sized candidate lists for I2T and T2I respectively. As illustrated in Fig. 3 and Fig. 4, our model achieves better performance than other approaches. The above experimental results on Pascal-Sentence, MIRFlickr, and NUS-WIDE indicate that CKD proposed in this paper is effective for cross-modal retrieval.

4.5 Discussion

4.5.1 Comparison with several classic subspace learning methods

In our experiments, we perform some comparative experiments to validate the performance of the proposed CKD. On three standard benchmark datasets, our method outperforms several classic cross-modal subspace learning methods. CCA learns a latent common subspace by maximizing the pairwise correlations between two modalities. KCCA incorporates a kernel mapping based on CCA. ml-CCA, an extension of CCA, takes into account the semantic information in the form of multi-label annotations to learn shared subspace. Many CCA-like methods and CCA-based variants learn common subspace representation by maximizing the feature correlations between multi-modal data. Unlike these methods, KDM learns subspace representation by maximizing the kernel dependency. The performance of KDM is superior to CCA, KCCA, and ml-CCA which illustrates that the consistency between feature-similarity and semantic-similarity can help to learn more discriminative feature representation for cross-modal retrieval. Although KDM preserves the consistency of the similarity among samples for each modal, it does not ensure the discriminative feature representation in Hilbert space. To overcome this problem, the proposed CKD uses the label information to construct a semantic graph and hopes that the learned subspace representation can preserve the semantic structure. The framework of CKD includes two parts: kernel Correlation Maximization and Discriminative structure-preserving. To validate the contribution of each part to the final retrieval results, we conduct some ablation experiments on three benchmark datasets. KDM learns a subspace by maximizing the kernel correlation between multi-modal data, while CKD($\beta=0$) only considers preserving the semantic structure for each modal in the process of learning a common space. Obviously, CKD is superior to KDM and CKD($\beta=0$) in terms of retrieval precision, which manifests that the proposed CKD integrating the kernel correlation maximization and the discriminative structure-preserving can improve the retrieval performance.

4.5.2 Performance Comparison of our model based on DNN features

Deep neural networks have demonstrated their powerful ability to encode effective feature representation and have been successfully applied to the information retrieval field. In this section, we replace the original features of our model with the DNN features for the multi-modal input data, which is termed as DCKD. Specifically, the 4096 dimensional CNN visual features are extracted by the fc7 layer of VGGNet which is pre-trained on ImageNet. We introduce the word2vector neural network to generate a 300-dimensional vector to represent each text. The NUS-WIDE dataset has been widely used as a benchmark dataset on cross-modal retrieval by many researchers. We compare our DCKD method with 7 deep learning approaches on this dataset. The compared methods based on DNN include MultimodalDBN [46], Bimodal-AE [47], Corr-AE [48], DCCA [29], CMDN [49], ACMR [30], Deep-SM [31]. Table 3 shows the MAP results of our DCKD and the compared methods on NUS-WIDE. As shown in Table 3, we can see that our DCKD outperforms all compared methods. In Table 3, we can observe the following points: (1) Our method that leverages label information to model the intra-modal structure and the inter-modal correlation is very effective. (2) The performance of our model is improved significantly when using DNN features.

4.6 Parameter sensitivity analysis

In this section, we explore the impact of the parameters involved in the proposed model on retrieval precision. As formulated in (7), the two parameters $\alpha_{1}$ and $\alpha_{2}$ control the weight of two modalities respectively. We observe the performance variation by tuning the value of $\alpha_{1}$ and $\alpha_{2}$ in the range of { 1e-5,1e-4,1e-3,1e-2,1e-1,1,10,1e2,1e3,1e4,1e5 }. Fig. 5 plots the performance of CKD I2T and T2I as a function of $\alpha_{1}$ and $\alpha_{2}$. As shown in Fig.5, we can see that our model on Pascal-Sentence is more sensitive to $\alpha_{1}$ and $\alpha_{2}$ than on MIRFlickr and NUS-WIDE. In addition, for the dimension $d$ of Hilbert space, we carry out experiments on Pascal-Sentence, MIRFlickr, and NUS-WIDE by changing the value of $d$ in the range of {10,20,30,40,50,60}. Fig.6 illustrates the MAP curve when $d$ changes in the candidate range, which reveals that CKD achieves the best performance when $d$ is set as 50,50 and 60 on Pascal-Sentence, MIRFlickr and NUS-WIDE respectively.

4.7 Complexity analysis

In this section, we discuss the complexity of the proposed CKD. The time complexity of Algorithm 1 is mainly on updating $P_{1}$ and $P_{2}$ by the eigenvalue decomposition on $Q_{1}$ and $Q_{2}$ respectively. In each iteration, the eigenvalue decomposition on $Q_{v}$ ($v=1,2$) costs $\mathcal{O}(d_{v}^{3})$. If the algorithm converges after $t$ iterations, the total complexity of our model is $\mathcal{O}(td^{3})$, where $d=\max(d_{1},d_{2})$. The cost of the training stage does not grow significantly with the increase in the size of the training set, which is flexible and adaptive enough for efficient large-scale retrieval tasks. Furthermore, we investigate the running time of the proposed CKD and other methods. Some experiments are conducted on Pascal-Sentence dataset to analyze the time consumption of the training and testing stage. The experimental results are summarized in Table 4. As shown in Table 4, we can observe that: (1) the running time of CKD is lower than ml-CCA whose complexity is $\mathcal{O}(n^{2}d^{2}+d^{3})$, where $n$ is the number of the training set. (2) In the testing stage, the proposed CKD is much more efficiently than the other approaches, which is very fast to search for information from the database.

5 Conclusion

In this paper, we present a novel method that integrates kernel correlation maximization and discriminative structure-preserving into a joint optimization framework. A shared semantic graph is constructed to make the subspace representation preserve the semantically structural information among data. Our model with the multiple supervision information facilitates to learn discriminative subspace representation for cross-modal retrieval. The experimental results on three publicly available datasets show that our approach is effective and outperforms several classic subspace learning algorithms.

Acknowledgements.

THE PAPER IS SUPPORTED BY THE NATIONAL NATURAL SCIENCE FOUNDATION OF CHINA(GRANT NO.61672265,U1836218), MURI/EPSRC/DSTL GRANT EP/R018456/1, AND THE 111 PROJECT OF MINISTRY OF EDUCATION OF CHINA (GRANT NO. B12018).