Bilinear Supervised Hashing Based on 2D Image Features

TL;DR

This paper introduces a bilinear supervised hashing method that directly utilizes 2D image features to generate binary codes, effectively preserving image structure and improving retrieval accuracy.

Contribution

It proposes a novel BSDH approach that leverages bilinear projections and discrete optimization to enhance binary code quality from 2D image features.

Findings

Outperforms existing hashing methods on benchmark datasets

Effectively preserves 2D image structure in binary codes

Improves retrieval performance with different image features

Abstract

Hashing has been recognized as an efficient representation learning method to effectively handle big data due to its low computational complexity and memory cost. Most of the existing hashing methods focus on learning the low-dimensional vectorized binary features based on the high-dimensional raw vectorized features. However, studies on how to obtain preferable binary codes from the original 2D image features for retrieval is very limited. This paper proposes a bilinear supervised discrete hashing (BSDH) method based on 2D image features which utilizes bilinear projections to binarize the image matrix features such that the intrinsic characteristics in the 2D image space are preserved in the learned binary codes. Meanwhile, the bilinear projection approximation and vectorization binary codes regression are seamlessly integrated together to formulate the final robust learning framework. Furthermore, a discrete optimization strategy is developed to alternatively update each variable for obtaining the high-quality binary codes. In addition, two 2D image features, traditional SURF-based FVLAD feature and CNN-based AlexConv5 feature are designed for further improving the performance of the proposed BSDH method. Results of extensive experiments conducted on four benchmark datasets show that the proposed BSDH method almost outperforms all competing hashing methods with different input features by different evaluation protocols.

Tables

Table 1. TABLE I: Notations Introduction

Notation	Size	Description
$n$	-	Number of samples
$c$	-	Length of hash code
$l$	-	Label dimension
$c_1$, $c_2$	-	Transition sizes for bilinear projection
$d_1$, $d_2$	-	Input sizes
$X_i$	$d_1\times d_2$	Input feature matrix for $i^{th}$ image
$B={\left\{b_i\right\}}_n$	$c\times n$	Hash code matrix
$Q_1$, $Q_2$	$d_1\times c_1$, $d_2\times c_2$	Bilinear projection matrix used in BSDH
$Y={\left\{y_i\right\}}_n$	$l\times n$	Sparse label matrix for database

Algorithm 1. Learning Algorithm for BSDH
Input: training data $\{X_1,X_2,\dots X_n\}$ ; label matrix $Y$; binary code length $c$; Transition sizes $c_1$ and $c_2$. maximum iteration number $t_1$, $t_2$; hyper-parameters $\lambda$, $\mu$;
Output: binary codes $B$; classification projection matrix $W$, bilinear projection matrices $Q_1$ and $Q_2$.
1. Compute the mean of dataset $M_0$ and the mean of $i\text{th}$ class$M_i$. Initialize bilinear projection matrix $Q_2$ as identity matrix.
2. Iteratively update $Q_2$ and $Q_1$ iteratively by solving the eigen decomposition problem of ${\left(S_w^{Q_2}\right)}^{-1}{S}_B^{Q_2}$ and ${\left(S_w^{Q_1}\right)}^{-1}{S}_B^{Q_1}$, repeat for $t_1$ times.
3. Calculate $H$ by equation (15) and initialize $B$ as ${\{-1,1\}}^{c\times n}$ randomly.
4. Repeat the following steps until converge or reach maximum iterations $t_2$: • calculate $W$ by equation (24). • compute $U$ by equation (26). • learn $B$ bit by bit using DCC descent method by equation (38).

Table 3. TABLE II: Input feature setting for different datasets

Datasets	Feature type	Feature size
MNIST	Raw pixel	28 $\times$ 28
PASCAL_VOC	FVLAD	100 $\times$ 64
MirFlickr	FVLAD	100 $\times$ 64
CIFAR-10	AlexConv5	256 $\times$ 36

Table 4. TABLE III: MAP result on MNIST

Code Length	16	32	64	128
LSH	0.182	0.263	0.295	0.326
PCAH	0.255	0.224	0.198	0.174
SH	0.277	0.264	0.248	0.260
DSH	0.302	0.314	0.371	0.390
ITQ	0.387	0.427	0.451	0.463
BDAH	0.483	0.541	0.568	0.581
LFH	0.542	0.541	0.568	0.581
KSH	0.806	0.795	0.838	0.834
BSDH	0.844	0.878	0.888	0.892

Table 5. TABLE IV: MAP results on PASCAL

Code Length	16	32	64	128
LSH	0.180	0.180	0.181	0.181
SH	0.188	0.185	0.183	0.186
PCAH	0.184	0.182	0.181	0.179
DSH	0.183	0.188	0.191	0.189
ITQ	0.189	0.187	0.184	0.181
BDAH	0.193	0.194	0.194	0.199
KSH	0.248	0.194	0.184	0.184
LFH	0.273	0.271	0.273	0.262
BSDH	0.363	0.360	0.363	0.384

Table 6. TABLE V: MAP results on MirFlickr

Code Length	16	32	64	128
LSH	0.543	0.545	0.546	0.547
SH	0.552	0.551	0.548	0.548
PCAH	0.546	0.546	0.545	0.545
DSH	0.551	0.552	0.555	0.556
ITQ	0.539	0.541	0.544	0.549
BDAH	0.551	0.553	0.553	0.552
KSH	0.544	0.545	0.546	0.548
LFH	0.578	0.579	0.582	0.585
BSDH	0.635	0.656	0.672	0.689

Table 7. TABLE VI: MAP results for image retrieval using FVLAD and VLAD features on PASCAL by different methods

Methods	16		32
	VLAD	FVLAD	VLAD	FVLAD
SH	0.180	0.185	0.181	0.183
DSH	0.179	0.188	0.181	0.191
BSDH	0.311	0.360	0.340	0.363

Table 8. TABLE VII: MAP results of BSDH with AlexConv5 features and traditional hashing baselines with CNN features on CIFAR-10

Code Length	12	24	48
LSH	0.131	0.162	0.203
ITQ	0.271	0.283	0.291
PCAH	0.206	0.187	0.170
SH	0.194	0.192	0.175
LDAH	0.219	0.179	0.159
BRE	0.255	0.267	0.286
MLH	0.174	0.168	0.174
KSH	0.344	0.462	0.548
BSDH	0.474	0.756	0.773

Table 9. TABLE VIII: MAP results of BSDH and traditional hashing baselines both with AlexConv5 features on CIFAR-10

Code Length	12	24	48
LSH	0.114	0.115	0.119
ITQ	0.160	0.174	0.178
PCAH	0.147	0.140	0.132
SH	0.150	0.142	0.138
KSH	0.366	0.413	0.436
BSDH	0.474	0.756	0.773

Table 10. TABLE IX: MAP results of BSDH and traditional hashing baselines with handcraft features and CNN features on CIFAR-10

Methods	Handcraft feature		CNN feature
	16	32	16	32
LSH	0.121	0.138	0.135	0.175
ITQ	0.152	0.160	0.275	0.286
BRE	0.130	0.136	0.263	0.280
MLH	0.137	0.133	0.181	0.180
KSH	0.219	0.208	0.395	0.503
DeepHash	0.216	0.230	0.547	0.567
BSDH	0.278	0.299	0.689	0.754

Table 11. TABLE X: MAP results of BSDH with AlexCon5 features and deep hashing baselines on CIFAR-10

Code Length	24	32	48	64
CNNH	0.511	0.542	0.522	-
CNNH+	0.521	0.521	0.532	-
DNNH	0.566	0.558	0.581	-
DHN	0.594	0.603	0.621	-
DSH(2)	0.651	0.659	0.662	0.671
DQN	0.558	0.564	0.580	-
VDSH	0.541	0.545	0.548	-
DPSH	0.727	0.732	0.778	0.776
DeepHash*	0.729	0.735	0.741	0.748
DSCH*	0.613	0.617	0.620	0.624
DRSCH*	0.622	0.629	0.631	0.633
BSDH*	0.756	0.754	0.773	0.777

Equations

$$X\to R_1^{T}X{R}_2,$$

$$H\left(X\right)=\mathrm{vec}\left(\mathrm{sgn}\left(R_1^{T}X{R}_2\right)\right),$$

$$\sum _{i=1}^n\cos ({\theta }_i)=\sum _{i=1}^n\left(\frac{\mathrm{vec}{\left(\mathrm{sgn}\left(R_1^T{X}_i{R}_2\right)\right)}^T}{\sqrt{d}}\mathrm{vec}\left(R_1^T{X}_i{R}_2\right)\right)$$

$$=\frac{1}{\sqrt{d}}\sum _1^n\mathrm{tr}\left(B_i{R}_2^T{X}_i^T{R}_1\right).$$

$$Q\left(B,R_1,R_2\right)=\underset{B,R_1,R_2}{\max }\sum _{i=1}^n\mathrm{tr}\left(B_i{R}_2^T{X}_i^T{R}_1\right)$$

$$\begin{array}{cc}s.t.& B_i\end{array}\in {\left\{-1,+1\right\}}^{d_1\times d_2},R_1^T{R}_1=I,R_2^T{R}_2=I.$$

$$\underset{B,W,F}{\min }\sum _{i=1}^{n}L\left(y_i,W^T{b}_i\right)+\lambda {\Vert W\Vert }^2+\upsilon \sum _{i=1}^n{\Vert b_i-F\left(x_i\right)\Vert }^2$$

$$s.t.b_i=sgn\left(F\left(x_i\right)\right),i=1,...,n.$$

$$F\left(X\right)=P^T\varphi (X),$$

$$\varphi \left(X\right)={\left[\exp \left({\Vert X-a_1\Vert }^2/\sigma \right),\dots ,\exp \left({\Vert X-a_m\Vert }^2/\sigma \right)\right]}^T,$$

$$D_b$$

$$=tr\left(\sum _{i=1}^l{a}_i{Q_1}^T\left(M_i-M_0\right)Q_2{Q}_2^T{\left(M_i-M_0\right)}^T{Q}_1\right),$$

$$D_w=\sum _{i=1}^l\sum _{X_j\in {\Pi }_i}{\Vert {Q_1}^T\left(X_j-M_i\right)Q_2\Vert }^2$$

$$=tr\left(\sum _{i=1}^l\sum _{X_j\in {\Pi }_i}{Q_1}^T\left(X_j-M_i\right)Q_2{Q_2}^T{\left(X_j-M_i\right)}^T{Q}_1\right),$$

$$D_b=tr\left({Q_1}^T{S}_b^{Q_2}{Q}_1\right),$$

$$D_w=tr\left({Q_1}^T{S}_w^{Q_2}{Q}_1\right),$$

$$S_b^{Q_2}=\sum _{i=1}^l{a}_i\left(M_i-M_0\right)Q_2{Q}_2{\left(M_i-M_0\right)}^T,$$

$$S_w^{Q_2}=\sum _{i=1}^l\sum _{X_j\in {\Pi }_i}\left(X_j-M_i\right)Q_2{Q_2}^T{\left(X_j-M_i\right)}^T.$$

$$\max \left({\left(D_w\right)}^{-1}{D}_b\right)$$

$$=$$

$$h_i=\mathrm{vec}\left(Q_1^T{X}_i{Q}_2\right),$$

$$H=[h_1,h_2,\dots ,h_n].$$

$$\underset{B,U}{\min }{\Vert B-UH\Vert }^2$$

$$s.t.B\in {\left\{-1,1\right\}}^{c\times n}.$$

$$\underset{W,B}{\min }{\Vert Y-W^{T}B\Vert }^2+\lambda {\Vert W\Vert }^2$$

$$\begin{array}{cc}s.t.& B\in {\left\{-1,1\right\}}^{c\times n},\end{array}$$

$$\underset{B,W,U}{\min }{\Vert Y-W^{T}B\Vert }^2+\lambda {\Vert W\Vert }^2+\mu {\Vert B-UH\Vert }^2$$

$$s.t.B\in {\left\{-1,1\right\}}^{c\times n},$$

$$\underset{B,W,U}{\min }{\Vert Y-W^{T}B\Vert }^2+\lambda {\Vert W\Vert }^2$$

$$\iff$$

$$+\mathrm{tr}(B^{T}W{W}^{T}B)+\lambda \mathrm{tr}(W^{T}W).$$

$$N(W)$$

$$+\mathrm{tr}(B^{T}W{W}^{T}B)+\lambda \mathrm{tr}(W^{T}W).$$

$$\underset{W}{\min }N(W)=\underset{W}{\min }\begin{array}{rl}& \mathrm{tr}(B^{T}WY)-\mathrm{tr}(Y^T{W}^{T}B)\\ & +\mathrm{tr}(B^{T}W{W}^{T}B)+\lambda \mathrm{tr}(W^{T}W).\end{array}$$

$$(B{B}^T+\lambda I)W=B{Y}^T.$$

$$W={(B{B}^T+\lambda I)}^{-1}B{Y}^T.$$

Full text

I Introduction

Hashing has been a hot research topic on image processing and pattern recognition in the past decade due to its promising performance in image and video storage and retrieval [1, 2, 3]. Hashing aims to find a series of hash functions to map the high-dimensional image features into a low-dimensional hamming space. The final purpose of hashing is to learn good binary codes that can preserve the similarity correlations and well represent images with the minimum information loss.

Learning to hash method is also known as data-dependent hash codes learning. It has become a substantial part of hashing research and attracted much attention owing to its promising performance [4, 5, 6, 7]. Its main objective is how to design the hashing function to minimize the distance (loss) between the low-dimensional hashing features (binary codes) and the original high-dimensional features, or how to preserve similarities between them. To this end, a number of distance and similarity measurements were proposed, including minimizing the product of the similarity and distance, maximizing the similarity-similarity product, minimizing the distance-distance difference, and etc. [8, 9, 10, 11]. Quantization error minimization is a widely-adopted strategy for objective function design as a special form of distance-distance difference minimization. Existing studies claim that using quantization method can obtain high-quality hash codes that can achieve better performance for image retrieval task.

Although a large number of hashing methods have been proposed with different objective functions, most of them are still based on linear projection and only focus on high-dimensional vectorized features [12]. Even though some 2D features are adopted, they are still manually converted into vectors in practice. However, such operation fail to capture the structural information contained in matrix-form features. Actually, natural matrix structures are frequently observed in image features such as vector of locally aggregated descriptors (VLAD), Fisher Vector and Histogram of oriented gradient (HOG) [13, 14, 15]. These features carry abundant information in matrix form. As introduced in Kim’s work [16], the residual between matrix features and their nearest visual vocabularies is encoded into a matrix form. As a result, manually vectorizing these matrices may lose structural information and weaken their performance. Moreover, when 2D matrix feature is re-organized as a high-dimensional vector feature, the high storage and time complexity cause a big problem for large projection construction. Therefore, a better choice is to develop 2D hashing function learning to take advantage of small size matrix-form features, resulting in better performance.

Bilinear models have been successfully used in other applications such as 2D-PCA, 2D-CCA, 2D-LDA [17, 18, 19] to directly process 2D inputs, the results of which showed that better performance can be achieved via bilinear learning. Besides, the well-known convolutional neural network (CNN) has shown exceptional performance on image analysis tasks. The advancement, to most degree, benefits from the convolutional layers where the convolution operation can capture local dependencies in the original image and preserve the spatial relationship between pixels. The success of CNN and its application in different areas shows the importance of containing spatial information in image analysis, and also indicates that it is promising in hashing to obtain high-quality binary features if a suitable objective function can be designed to take the advantages of the matrix form features.

Recently, a few works have paid attention to 2D image features and some bilinear hash learning methods were proposed [16, 20, 21]. One representative work [21] is the bilinear projection based binary codes (BPBC) learning method, which formulates the objective function by minimizing the angle between the rotated feature vector and its binary codes. Specifically, two rotation matrices are employed to rotate descriptor matrix for generating effective binary codes. However, since BPBC is an unsupervised method, the label information is not considered, yielding inferior retrieval performance. Moreover, since its main idea is to strengthen the original feature using rotation strategy, the performance is degraded when the binary codes are short. Similarly, other methods [16, 20] also did not achieve very satisfactory results either.

Due to the inspiration and limitation of the current research mentioned above, in this paper, we propose a bilinear supervised discrete hashing method (BSDH) based on quantization loss minimizing to learn binary codes from 2D image features. This method utilizes bilinear projection to binarize matrix features which can keep the image matrix’s intrinsic space relationship and integrates the idea of bilinear projection approximation and vectorization binary codes regression. The main contributions of this paper are summarized as follows:

1. We propose a 2D feature-based hashing method in which the design of the objective function is based on matrices instead of vectors. This design can effectively avoid the loss of crucial spatial information of images contained in 2D features. The proposed method integrates the bilinear projection approximation and supervised linear regression, so that high-quality binary codes are promised to be learned.

2. A novel optimization strategy based on bilinear discriminant analysis and discrete optimization is developed. The discrete optimization strategy is an alternatively iterative method, which minimizes the quantization loss of the learned binary codes and the projected real value matrix in the regression to generate high-quality binary codes.

3. We explore two kinds of feature extraction methods to extract 2D image features for the proposed hashing method, i.e., non-deep FVLAD (Fast VLAD) feature and deep AlexConv5 feature. The FVLAD feature is designed based on SURF (speed up robust feature) feature as an efficient alternative to VLAD, while the AlexConv5 feature is extracted from the last convolutional layer of the typical CNN, i.e., AlexNet. According to experimental results, the proposed BSDH obtains higher recall rates and MAPs against other methods. Further, with AlexConv5, the performance of BSDH is significantly enhanced.

The rest of this paper is organized as follows. In section II, related studies about supervised discrete hashing and bilinear projection binary codes learning are briefly reviewed. In section III, our bilinear supervised discrete hash learning model is proposed. Section IV introduces the discrete optimization algorithm we adopted to solve the optimization model. Experiments are explained in section V to evaluate the performance of BSDH. Finally, conclusions are given in section VI.

II related work

In this section, we briefly review some related studies about bilinear projection based binary codes and supervised discrete hashing.

II-A Bilinear Projection Based Binary Codes

The bilinear projection based binary codes (BPBC) method is a bilinear binary code learning method which explores the feature enhancement for better retrieval and classification performance using the matrix rotation strategy. Specifically, the high-quality binary codes are obtained by bilinear rotating the original feature matrix without dimensionality reduction with two random orthogonal matrices as follows:

[TABLE]

where $X$ is the original data and ${{R}_{1}}$, ${{R}_{2}}$ are two rotation matrices. Combined with threshold and vectorization, the final bilinear code is generated by:

[TABLE]

where $\operatorname{sgn}\left(\centerdot\right)$ is the sign function and $\operatorname{vec}\left(\centerdot\right)$ denotes vectorization. To learn the binary codes, the authors chose to minimize the angle ($\theta$) between the rotated feature vector $\operatorname{vec}\left(R_{1}^{T}X{{R}_{2}}\right)$ and its binary encoding $\operatorname{vec}\left(\operatorname{sgn}\left(R_{1}^{T}X{{R}_{2}}\right)\right)$:

[TABLE]

The final objective function turns into:

[TABLE]

The objective function can be easily solved by alternatively updating variables. Specifically, ${{R}_{1}}$ and ${{R}_{2}}$ are obtained by singular value decomposition (SVD), and low dimensional binary codes can be learned if ${{R}_{1}}$ and ${{R}_{2}}$ are downsized. However, the bilinear rotation discussed in this research is without dimension reduction. Even though authors considered learning low-dimensional binary codes, they only explored the performance of hash codes with half dimension as the original feature, while the learned binary codes are still highly dimensional. Based on our experimental results, its performance was degraded with the decrease of the binary code length, which is probably caused by not using any label information.

II-B Supervised Discrete Hashing

Supervised discrete hashing (SDH) [22] attempts to use the manifold embedding strategy for formulating the hashing framework that can jointly learn a binary embedding and a linear classifier. Assuming that ${{x}_{i}}$ is the input descriptor vector, ${{y}_{i}}$ and ${{b}_{i}}$ are the label vector and binary codes respectively, $W$ denotes the mapping matrix for regression. The objective function of SDH is composed of ridge regression loss and embedding fitting error as follows:

[TABLE]

The embedding function here is:

[TABLE]

where $\phi\left(x\right)$ is nonlinear anchor feature vector obtained by the RBF:

[TABLE]

where ${{a}_{i}}$ and $\sigma$ are anchor points and kernel width. The kernel trick used here was widely used in kernel hash function e.g., BRE [8] and KSH [23, 24]. It has been proven that using kernel strategy is helpful to improve retrieval precision, but the computational cost will also increase. To solve the optimization problem in SDH, researchers kept the binary constraint and used discrete strategy to iteratively update variables. For the most challenging part in learning $B$, they kept the binary constraint and used the novel discrete cyclic coordinate descent method.

This method formulated a joint objective function which leveraged label information and used discrete optimization method to solve the optimization problem. However, it only focused on vectorized features which cannot exploit the important structural information contained in 2D image features. Besides, the kernel embedding was time consuming to some extent, both effectiveness and efficiency were not promising.

III Bilinear supervised discrete hashing method

In this section, we firstly introduce how to employ the bilinear projection to obtain the low-dimensional representation of 2D image features. Then the supervised hashing stream is introduced to learn high-quality hash codes.

III-A Motivation and General Idea

The hashing algorithm proposed in this paper is mainly inspired from two aspects. Firstly, as illustrated in section I, bilinear models show great potential to be used to develop well-performed hashing algorithm as they can directly deal with 2D features. It can effectively capture structure information contained in 2D image features and can avoid memory problem in calculation when applied on very big image datasets. However, existing bilinear hashing methods did not achieve remarkable performance since their objective functions were not well designed. For example, for BPBC, the objective function solely focused on minimizing the quantitation loss of the thresholding, which means that it only used bilinear mapping to make rotation of ordinary data for binary approximation, without exploiting any labels information and reconstruction information. Therefore, it is hard to obtain high-quality codes. Secondly, as indicated in section II, the SDH achieved good performance in integrating binary codes learning and image classification, which demonstrates that minimizing quantization loss and using discrete optimization strategy are effective. However, it is not efficient enough as the kernel embedding used for binary coding is time consuming and it becomes an obstacle for its application in big data scenarios. More importantly, the SDH method is based on vectorized features instead of 2D features. According to above discussion on the advantages of 2D features, applying 2D features is possible to further improve its performance.

Based on above consideration, we propose a novel bilinear supervised discrete hashing (BSDH) method, which integrates the bilinear projection and vectorization binary codes regression. The idea of BSDH is to learn a bilinear mapping based on original image feature matrix so as to obtain low-dimensional 2D features and then to learn the binary codes by solving the supervised linear regression problem. Label information and reconstructive information are used in the proposed objective function simultaneously. Since solely ignoring the binary constraint or substituting it with relaxed function may degrade the performance, we adopted the discrete regularization learning in optimizing the proposed model. As such, the model can obtain binary codes with high performance by the 2D image features.

Figure 1 illustrates the concept and principle of the proposed method. Firstly, we conduct bilinear projection on the raw image pixels or high-dimensional 2D features to obtain relatively low-dimensional 2D representation. Then we use the low-dimensional representation to further learn the compact binary code by supervised hash coding.

III-B Low-dimensional 2D Feature by Bilinear Projection

For ease of presentation and understanding, notations used in the following section are listed in Table I.

Similar to BPBC, we use two matrices to conduct bilinear projection on high-dimensional 2D feature ${{X}_{i}}$ as ${{Q}_{1}}^{T}{{X}_{i}}{{Q}_{2}}$, but the difference is that the matrices are in smaller sizes. With ${{Q}_{1}}\in{{R}^{{{d}_{1}}\times{{c}_{1}}}}$ and ${{Q}_{2}}\in{{R}^{{{d}_{2}}\times{{c}_{2}}}}$, the input feature can be downsized from ${{d}_{1}}\times{{d}_{2}}$ to ${{c}_{1}}\times{{c}_{2}}$. Here we name ${{c}_{1}}$ and ${{c}_{2}}$ transition sizes. Inspired by linear discriminant analysis (LDA) [19], we consider the inter-class scatter matrix ${{D}_{b}}$ and intra-class scatter matrix ${{D}_{w}}$ as follows:

[TABLE]

[TABLE]

where ${{a}_{i}}$ is the number of samples of ${{i}^{\text{th}}}$ class, ${{M}_{i}}$ and ${{M}_{0}}$ are the mean of input for ${{i}^{\text{th}}}$ class and the mean of input for the whole dataset. By simple formulation, ${{D}_{b}}$ and ${{D}_{w}}$ can be further transformed into:

[TABLE]

[TABLE]

where $S_{b}^{{{Q}_{2}}}$, $S_{w}^{{{Q}_{2}}}$ are defined as follows:

[TABLE]

[TABLE]

To make the bilinear projection with minimum information loss, we aim to maximize the between-class distance and minimize within-class distance, which equals to maximize the following function:

[TABLE]

where ${{Q}_{1}}$ can be obtained by executing eigen decomposition on the matrix ${{\left(S_{w}^{{{Q}_{2}}}\right)}^{-1}}S_{b}^{{{Q}_{2}}}$. Similarly, we can obtain ${{Q}_{2}}$ by solving the eigen decomposition problem of ${{\left(S_{w}^{{{Q}_{1}}}\right)}^{-1}}S_{b}^{{{Q}_{1}}}$. After obtaining ${{Q}_{1}}$ and ${{Q}_{2}}$ , we can calculate the bilinear-projected matrix $H$ for further updating of other variables by the following equations.

[TABLE]

[TABLE]

III-C Bilinear Supervised Discrete Hashing

To achieve better binary fitting based on the obtained bilinear feature matrix $H$, we further impose the linear projection on the matrix and minimize the quantity loss for the binary codes as follows:

[TABLE]

To take the advantage of label information to enhance the performance, we further introduce the linear regression with the label information and binary code in the objective function. The rational of the objective function is that good binary codes should be able to have minimum regression loss. Then we have:

[TABLE]

where $W$ is the classification projection matrix, $Y=\left\{{{y}_{1}},{{y}_{1}}\cdots{{y}_{n}}\right\}\in{{R}^{l\times n}}$ is labels matrix consisting of label vectors of pictures and $l$ is the length of label vector. By combining the equations Eqn.(17) and Eqn.(18), the final objective function of our BSDH is formulated as

[TABLE]

where $H=[\operatorname{vec}(Q_{1}^{T}{{X}_{1}}{{Q}_{2}}),...,\operatorname{vec}(Q_{1}^{T}{{X}_{n}}{{Q}_{2}})]$.

IV Optimization Algorithm

There are three variables to be updated in solving problem (19). The most challenging task comes from the binary constraint of $B$, which leads to an NP-hard problem. To this end, the existing methods directly drop the binary constraint or substitute it with continuous functions. However, both strategies may cause large quantity loss [25], resulting in degraded performance. Based on recent research results [22, 26], we propose a discrete optimization to address the resulting optimization problem (19) instead of using any relaxing scheme. We employ an alternating optimization algorithm to iteratively update variables, i.e. $B$, $W$ and $U$. That is, we optimize each variable when fixing others.

Updating $W$ with others fixed: Taking $W$ as the variable while the other variables being fixed, the sub-objective function to solve this problem is:

[TABLE]

The first term in the above equation is constant. Considering the function ${{N}}(W)$ which consists of the last four terms:

[TABLE]

Then we aim to

[TABLE]

To solve problem (22), $W$ should satisfy the following condition:

[TABLE]

We can finally obtain $W$:

[TABLE]

Updating $U$ with others fixed: The objective function with respect to $U$ is:

[TABLE]

Take the derivation with respect to $U$ and make it equal to zero, we can obtain the solution as follows:

[TABLE]

Update $B$ with others fixed: Updating $B$ is the most challenging part for the whole optimizing process because we need to handle the binary constraints, i.e., the sgn function in this part. The sub-objective function for $B$ is

[TABLE]

We have the following optimization problem from Eqn.(27):

[TABLE]

[TABLE]

It is noted that tr$({{B}^{T}}B)=cn$ is constant and thus the formulation turned into:

[TABLE]

To solve problem (30), we use discrete cyclic coordinate (DCC) descent method to learn $B$ row by row, i.e., to learn ${{b}_{i}}$ bit by bit. Let ${{v}^{T}}$ be ${{l}^{th}}$ row of $W$ , ${W}^{\prime}$ be the rest matrix of $W$, and ${{z}^{T}}$ be ${{l}^{th}}$ row of $B$, ${B}^{\prime}$ be the rest matrix of $B$. Now we consider the first term of (30):

[TABLE]

When ${W}^{\prime}$, ${B}^{\prime}$ are fixed and $z$ is variable, for Eqn.(31) we have:

[TABLE]

[TABLE]

where $const$ denotes constant. Thus, only second term in Eqn.(31) is variable, and Eqn.(31) turns into:

[TABLE]

Now we consider the second and third term of Eqn.(30):

[TABLE]

Let $M={{Y}^{T}}{{W}^{T}}+\upsilon{{H}^{T}}{{U}^{T}}$, $m$ be the ${{l}^{th}}$ column of $M$ and ${M}^{\prime}$ be the rest matrix of $M$, Eqn.(35) can be reformed to:

[TABLE]

After above derivations, the objective function (30) turns into:

[TABLE]

The optimal solution of problem (37) is:

[TABLE]

In summary, the optimal solution is obtained by alternatively updating variables, and the whole optimization problem is divided into few steps. In the step of updating $B$, each bit is calculated based on the rest bits iteratively. The convergence of the proposed algorithm is promising. Experimental results show that the objective function value can be convergent within ten iterations. The detailed algorithm steps are presented in algorithm 1.

V Experiments

To evaluate the performance of the proposed BSDH method for image retrieval task, extensive experiments were conducted on four image datasets using different input features. The experimental results of BSDH were further compared with the results of existing hashing baselines, including traditional hashing methods and deep hashing methods.

V-A Datasets and Baselines

V-A1 Datasets

We evaluate the performance of the proposed method on four benchmark datasets, i.e. MNIST [27], CIFAR-10 [28], PASCAL_VOC and MirFlickr [29]. Specifically, images in MNIST and CIFAR-10 only have single label while images in the other two datasets have multiple labels. The ground-truth is defined based on labels, that is, images sharing at least one label are considered as similar. More detailed introduction of these datasets are as follows.

MNIST consists of 60,000 $28\times 28$ small grey-scale images of handwritten digits from ‘0’ to ‘9’ in the training set. We randomly chose 54,000 images as the training set and used the rest 6000 images for testing. For our method and other bilinear projection methods, the raw pixel matrix was used as input data. For other methods, the pixel matrix was vectorized to a 784-dimensional vector as the input.

PASCAL_VOC is offered for annual PASCAL_VOC challenge of image classification, detection and segmentation containing similar size images. We used VOC2012 in this paper and randomly chose 15,000 images of 20 classes in our experiment. All images were divided into two parts, i.e. 10k for training and 1500 for testing.

MirFlickr is a micro version of dataset MirFlickr-1M. This dataset consists of 25,000 images with 24 classes of objects. In our experiment, we labeled this dataset based on its original annotations. Specifically, each image in the dataset was labeled with a 24-dimensional vector corresponding to 24 classes of objects. For image allocation, we used 10000 images to train our hashing coding method and another 5000 images to test the hashing model.

CIFAR-10 consists of 60,000 $32\times 32$ tiny color images from 10 categories, each of which has 6,000 samples. We randomly selected 1,000 images from each category and constructed a test set with 10,000 images. The remaining 50,000 images were used as training data. The CIFAR-10 dataset was used for conducting comprehensive comparative experiments to evaluate the effectiveness of the proposed BSDH method, especially compared to some deep baselines and non-deep methods using deep features.

V-A2 Baselines

To demonstrate the effectiveness of the proposed method, our BSDH is compared against classic or state-of-the-art competing hashing schemes, including $11$ non-deep hashing methods and $11$ deep hashing methods. Specifically, non-deep hashing methods include ITQ [30], SKLSH [31], SH [32], DSH [33], PCAH [34], LSH [35], BDAH [20], KSH [23], LFH [36], BRE[8] and MLH [37], and deep hashing methods are CNNH [38], CNNH+ [38], DNNH [39], DHN [40], DSH(2)[41], DQN [42], VDSH [43], DPSH [44], DSRH [45], DSCH [46], DRSCH [46].

There are two hyper-parameters in the proposed algorithm, i.e., $\lambda$ and $\mu$. We employ the grid-search strategy to find the best parameters from the set $\{10^{-5},10^{-4},10^{-3},10^{-2},10^{-1},1,10\}$. In the experiments, we set $\lambda={{10}^{-5}}$ and $\mu={{10}^{-1}}$ for our method on all the datasets. For most baseline methods, we re-implemented the source codes provided by authors with default parameters. For those methods who unreleased codes such as BDAH, we wrote the codes by ourselves according to the algorithm stated in original papers. To make fair comparison, all experiments were conducted five times for all methods, and the averaged performance are reported.

V-B Experimental Setting

V-B1 Image feature extraction

An effective image feature extraction algorithm is foundation of many image analysis tasks. To incorporate proper image features into the proposed BSDH method and further investigate the influence of input features, we design two 2D feature extraction methods: a traditional one based on SURF [28] and a deep feature based on CNN [47] , which are introduced below in detail .

Traditional 2D feature - FVLAD: As we know, VLAD [13] feature is an effective 2D feature which is detectable even under changes in image scale, noise and illumination. It is constructed based on the SIFT (scale invariant feature transform) [48]. However, SIFT has the obvious drawback for its application, i.e., computational complexity. To avoid heavy computation, we design a descriptor based on SURF (speed up robust feature), which is much faster to calculate, almost three times faster than SIFT [49]. The new descriptor is similarly constructed as VLAD and named as FVLAD (Fast VLAD).

Deep 2D feature - AlexConv5: There is no doubt that convolutional neural network (CNN) has been the most powerful tool in computer vision. To further testify the effectiveness of our method, we designed a new 2D convolutional feature and evaluated the performance of the proposed method with this feature. Notably, the most widely used CNN features are extracted from the fully-connected (FC) layer [50]. However, such features cannot capture the spatial information, leading to inferior performance. Therefore, we chose to directly take the output of the convolutional layers as the input feature for the proposed BSDH method. Specifically, we adopted the AlexNet [47] pre-trained on ImageNet dataset as the CNN feature extractor and extracted the output of the last convolutional layer (5th conv layer) as the image feature, which is with the size of $256\times 36$ and named as AlexConv5.

V-B2 Evaluation protocols

When binary codes learning was finished, hamming distance was calculated to measure the difference between the query images and training images, and images with smaller difference to queries will be retrieved as target results. To evaluate the image retrieval performance, the commonly used mean average precision (MAP) [51], recall rate with the number of retrieved samples, precision rate with the number of retrieved sample and precision-recall curves are used as the criteria for performance evaluation. Considering the result vector

[TABLE]

for a query $q$ after hash ranking, the recall rate, precision rate and MAP are defined as follows [51]:

[TABLE]

[TABLE]

[TABLE]

where $rel\left({{v}_{i}}\right)=\left\{\begin{matrix}1&relevant\\ 0&irrelevant\\ \end{matrix}\right.$ denotes the relevance of the ${{v}_{i}}$.

V-C Experimental Results and Analysis

To comprehensively evaluate the performance of the proposed method, we conducted extensive experiments with different input features on different datasets. The detailed experimental layout is illustrated in Table II.

V-C1 Experimental results on MNIST (raw pixel features)

The MAP result, recall rate curve with retrieved number, precision rate curve with retrieved number and precision-recall curve of the proposed method and baselines on MNIST are reported in this section. MAP results are shown in Table III.

It is clear in Table III that our method outperforms all other compared method in MAP results. It is worth noting that among all comparison methods, BDAH also adopts the strategy of bilinear projection, but it only achieves the MAP around 0.5. With the similar idea of bilinear projection, the proposed BSDH method achieves the MAP over 0.8. The reason may be that our method integrates linear regression in the code learning and fully takes advantage of the label information. Further, compared with supervised hashing methods, namely BDAH, KSH, LFH, our method shows best performance in all configurations. As all compared methods learn hash codes under the supervision of label information, the main reason for BSDH obtaining better performance is probably because it directly conducts bilinear projection on pixel matrix and does not change it to vector manually. According to previous research, the matrix input contains abundant structural information about spatial relationship between pixels, which will be destroyed if treated as vectors [16].

Figure 2-4 illustrate the recall rate, precision rate and recall-precision curve with respect to different code length from 8 to 128. It is clear that the BSDH method almost outperforms all compared methods on all three criterions, especially for recall-precision curves where it shows superior performance. It can also be seen that our method shows consistently good performance on the cases of different code lengths, except for very short code of 8-bits. Comparatively, one baseline method LFH can also achieve good results when the code length is long, while for short code cases, its performance is apparently degraded.

V-C2 Experimental results on PASCAL and MirFlickr (FVLAD features)

Table IV and Table V report the MAP results of comparison experiments on PASCAL and MirFlickr respectively. In this section, we compared our method with other seven methods, including five unsupervised methods and two supervised methods. From Table IV, we can see that BSDH still achieves good results on PASCAL. Specifically, it outperformed all other compared baselines in different code length and it is the only method achieved results over 0.3 on this dataset. It is worth noting that the overall performance on this dataset is not good. The possible reason may be that images in that dataset are not mostly correlative, which means that the similar pairs are fewer and the useful information for learning is limited. We also show the recall-precision results on PASCAL in Fig. 5. From the three figures, we can see that although BSDH does not achieve good performance when retrieving very few samples, it keeps the precision with the increasement of recall rate. This also explains why it can obtain better MAP results.

From Table V, it is clear that our BSDH outperforms all the compared methods in all cases, which further demonstrates the superiority of the proposed bilinear learning framework. Another observation from Table V is that the overall results are much better than that of PASCAL. This is because the training images we used are labeled by rich information which makes the relative targets easier to recall.

We used the FVLAD feature as the inputs on both PASCAL and MirFlickr for the proposed BSDH and other baselines. The results of these datasets show that this feature can help achieve good retrieval performance, especially when integrating with our bilinear projection strategy. To further support this point, we conducted extra experiments on PASCAL to evaluate the effectiveness of the used FVLAD feature. Since FVLAD can be seen as a fast version of VLAD, we attracted VLAD feature on the database and conducted image retrieval experiments using three hashing methods (including the proposed BSDH method). The experimental results are shown in Table VI. Results in Table VI clearly show that the proposed FVLAD feature is very solid to be used as low-level image feature for hashing algorithms.

V-C3 Experimental results on CIFAR-10 (CNN-based features)

In this part, we discuss the performance of the proposed BSDH method with deep features by comparing it with deep hashing methods and traditional hashing methods using deep features. Most compared data in this part are from published papers and will be noted clearly in proper place. As mentioned, the CNN-based image feature, AlexConv5, is firstly extracted and then fed into BSDH algorithm for conducting standard image retrieval experiments. The MAP is also adopted in this part as evaluation criteria.

Firstly, we compared our BSDH method with traditional hashing methods in terms of the image retrieval performance with three lengths of hash code. All compared hashing methods adopted the 1024-d deep feature [52]. Similar to our AlexConv5 feature, this deep feature was also extracted from the last convolutional layer of CNN, but the network structures were slightly different in terms of the kernel size and other details. This feature was in vector-form and related detailed information can be found in Mu’s paper [52]. MAP results of BSDH and baselines are listed in Table VII, and most results of baselines are cited from the paper [52]. For our BSDH method, We conducted experiments with the same setting as Mu’s paper [52] except using different deep features.

From Table VII, we can see that the performances of most baselines do not improve much even when adopting deep image features. The MAPs for most methods are below 0.3 except for KSH, whose performance is better than others. However, the MAP results of KSH are still lower than 0.6. Compared with these baselines, the proposed BSDH method achieves markedly better MAP results. The MAPs of both 24-bit and 48-bit cases are more than 0.75, which are quite satisfactory. For 12-bit case, the result is not as good as the other two cases, but it is still the best compared with the other methods.

In order to eliminate the influence of different image features on comparison of our method and baselines, we applied the vectorized AlexConv5 feature as the input of five typical baselines and conducted additional experiments. Under this circumstance, the comparison between baselines and our method is completely fair. The experimental results are listed in Table VIII. It is clear that our BSDH method is still far better than others under this experimental condition, which firmly validates its’ effectiveness and shows the advancement of the BSDH method. In addition, comparing results from Table VIII and Table VII also show the influence of difference features on the performance of hashing methods. Generally, the change of input features does not affect much on compared baselines’ performance with respect to the MAP results.

In Table IX, more comprehensive comparisons are made between the proposed BSDH method and traditional baselines with both handcraft features and CNN-based features. It is noted that the handcraft feature used in our method is the gray pixel matrix and CNN feature is the proposed AlexConv5. For other compared methods, 800-d GIST feature and 1024-d CNN feature were adopted according to [52]. All comparable data are directly cited from Mu’s work [52].

From Table IX, we have following observations. Firstly, our BSDH method outperforms all baselines with both non-deep and deep features, and with deep feature, the advantage of the proposed BSDH method is more remarkable. Secondly, the improvements on performance resulting from deep features varies with different methods. For example, LSH and MLH are not benefited much from deep feature while ITQ, BRE and KSH show clearly better performance by using deep features. Among all baselines, the biggest improvement from altering input features is shown in DeepHash. Comparatively, by using CNN feature, the proposed BSDH method achieves the most significant improvement on its retrieval performance. Specifically, for the 16-bit case, the MAP increases from 0.278 to 0.689 (by 0.411). For the 32-bit case, it dramatically increases by around 0.45. The results show that our BSDH method may be suitable to handle deep features and is promising to achieve desired performance with deep image features. This is a great advantage of the BSDH method as nowadays there already exist many well-trained deep network models, deep features of images are not difficult to obtain, which makes the proposed method more suitable in real-world application.

Related research has shown that deep hashing methods achieved notable advancement. However recently it can be found in research papers that traditional non-deep hashing methods could have even better performance with the help of deep learning-based image features compared with integrated end-to-end deep hashing methods. As the experimental results illustrated in this paper above have proved that CNN feature can help improve performance of traditional hashing, in the following, we investigate if the proposed BSDH method can outperform any deep hashing methods using deep features. In Table X, we compare the performance of our method using AlexConv5 feature with 11 state-of-the-art end-to-end deep hashing methods. Considering the dramatic training time and computing resource cost of deep hashing methods, we did not reimplement those methods. The compared data in Table X are partly from original papers of those methods and partly from reference papers [48, 51, 53]. It is noticed that for the first eight deep hashing methods, the image number of the training set is 5000 as usual set in deep learning methods with the consideration of the computing cost. The rest three have same quantity of training data, database data and test data as our BSDH method.

From Table X, it is clear that the proposed BSDH method outperforms all compared deep hashing baselines with different hash code lengths. It is generally believed that end-to-end deep hashing methods usually achieve better performance because of good nonlinear fitting capability of deep neural networks [53]. However, to obtain desired performance, hours or even days are required to train the deep hashing model. In contrast, the proposed BSDH just takes a few minutes to learn hash codes, whose efficiency is much better. Importantly, with such small computational cost, the learned hash codes by BSDH are with desired representative ability considering the retrieval performance shown in Table X. In summary, compared with end-to-end deep hashing methods, the proposed BSDH method has the following advantages: 1) the training time is much less; 2) the input is flexible and desired performance can be achieved by using CNN features; 3) parameters in the model are much less such that time spent on parameter setting can be reduced. Comprehensively speaking, the proposed BSDH method shows advantages compared to existing deep hashing methods.

V-D Parameter Sensitivity Analysis

V-D1 Discussion on hyper-parameters

There are two hyper-parameters in the proposed algorithm, i.e., $\lambda$ and $\mu$. To analyze how the two parameters will affect the performance of the algorithm, we conducted experiments with different sets of values and made comparison on their performance on MNIST dataset. We tried 81 combinations and the results are shown in Fig. 6. As we can see, apparently the value of $\lambda$ has more influence on the results. When $\lambda$ is taken in the range of less than ${{10}^{-5}}$, the algorithm shows quite satisfactory performance. However, if $\lambda$ is relatively big, for instance, one or ten, the algorithm almost failed. Observing the objective function, we can see that $\lambda$ is the weight of the regularization part. When the weight is too big, the algorithm will be regularized so hard that its fitting ability is weaken. Therefore, $\lambda$ should be set comparatively small so that it can help prevent overfitting and at mean time not weaken overall performance of the algorithm. On the other hand, the proposed method is robust to the values of $\mu$, except when $\lambda$ is in the range of $[{{10}^{-4}},{{10}^{-2}}]$. This means that the quantization loss part in our objective function have more impact on the results when $\lambda\in[{{10}^{-4}},{{10}^{-2}}]$. Meanwhile, when $\lambda$ is too small or too large, the quantization loss part has less effect.

V-D2 Discussion on binary code length and transition size

In this part, we make discussion on the influence of the transition sizes (${{c}_{1}}$ and ${{c}_{2}}$) and the hash code lengths ($c$) on the performance of the proposed algorithm. We chose 12 groups of ${{c}_{1}}$ and ${{c}_{2}}$, and five code lengths to conduct image retrieval experiments on CIFAR-10, with the input feature as the proposed $256\times 36$ AlexConv5. In total, 60 sets of MAP results are illustrated in Fig. 7. From Fig 7, we can see that the performance is generally growing with the increasement of code length. The improvement is remarkable from 12-bit to 16-bit and becomes small for hash code more than 16-bit.

As for different choices of ${{c}_{1}}$ and ${{c}_{2}}$, Figure 7 shows that the performance of the proposed method changes with the change of ${{c}_{1}}$ or ${{c}_{2}}$. However, the changes are not big except for one case with ${{c}_{1}}=16$ and ${{c}_{2}}=256$, whose performance is very bad. This exceptional bad result may result from the too large difference between ${{c}_{1}}$ and ${{c}_{2}}$, where ${{c}_{2}}$ is 8 times as ${{c}_{1}}$. Another observation is that the performance is still satisfactory when ${{c}_{1}}$ and ${{c}_{2}}$ are relatively small. This is very important as the smaller ${{c}_{1}}$ and ${{c}_{2}}$ be, the computational complexity of the following part of the algorithm can be reduced more. This also means that the efficiency the algorithm can be improved without big sacrifice of performance by choosing relatively smaller ${{c}_{1}}$ and ${{c}_{2}}$. It is also noticing that in all cases ${{c}_{2}}$ is set as multiples of ${{c}_{1}}$ considering the size of input feature.

From the above discussion on parameters, we can see that our BSDH method is comparatively easy to achieve desired performance by slightly adjusting parameters. This is regarded as another merit of the proposed method as this method is not only effective, but also very robust with parameters.

VI Conclusion

In this paper, we propose a bilinear supervised hashing method named BSDH. The BSDH method is designed based on the idea of directly dealing with native matrix-form image features. It utilizes bilinear projections to directly process the image matrix features, which keeps the image matrix’s intrinsic space relationship and integrates the idea of bilinear projection approximation with vectorization binary codes regression in the objective function. The approach of bilinear projection is also helpful for saving computing memory which can better deal with high dimensional image features than commonly used single linear projection approach. The optimal solutions are obtained by using iteratively updating variables referring to the idea of bilinear discriminant analysis and discrete optimization strategy. Experimental results on four image datasets with handcraft and CNN features demonstrate the effectiveness and superiority of the BSDH methods.