Ordinal Distance Metric Learning with MDS for Image Ranking

TL;DR

This paper introduces an improved linear ordinal distance metric learning method using multidimensional scaling for image ranking, enhancing speed and accuracy over previous models.

Contribution

It proposes a novel approach combining linear metric learning with MDS and least squares fitting, maintaining data structure and improving ranking performance.

Findings

Better speed compared to previous models

Enhanced ranking accuracy

Maintains local data structures

Abstract

Image ranking is to rank images based on some known ranked images. In this paper, we propose an improved linear ordinal distance metric learning approach based on the linear distance metric learning model. By decomposing the distance metric $A$ as $L^TL$, the problem can be cast as looking for a linear map between two sets of points in different spaces, meanwhile maintaining some data structures. The ordinal relation of the labels can be maintained via classical multidimensional scaling, a popular tool for dimension reduction in statistics. A least squares fitting term is then introduced to the cost function, which can also maintain the local data structure. The resulting model is an unconstrained problem, and can better fit the data structure. Extensive numerical results demonstrate the improvement of the new approach over the linear distance metric learning model both in speed and ranking performance.

Tables

Table 1. Table 1: Computational Complexity for Algorithm 2 and LDMLR.

Algorithm 2		LDMLR
Step	Complexity	Complexity	Step
S0	$O\left(sd\right)$	$O\left(d^2\right)$	Initialize
S1	$O\left(n^2\right)$	$O\left(d{n}^2+K{n}^2\right)$	$K$ target
S2	$O\left(n^3\right)$		neighbor
S3	$O\left(d{n}^2+K{n}^2\right)$		search
S4	$O\left(nsd+nk\left(d+sd+s^2\right)\right)$	$O\left(n^2{d}^2+nK{d}^3\right)$	$\underset{¯}{\nabla h\left(A\right)}$
S5	$O\left(r\left(nKsd+nK{d}^2\right)\right)$	$O\left(d^3\right)$	$\underset{¯}{{\Pi }_{S_{+}^n}(\cdot )}$
S6	$O\left(ds\right)$

Table 2. Table 2: Results of cMDS-DML on the UMIST dataset, with different values of dimension s 𝑠 s .

$s$	2	3	5	8	10
MAE	0.3539	0.3463	0.3498	0.3684	0.3798
STD	0.0812	0.0671	0.0640	0.0762	0.0830
t(s)	2.27	3.10	4.58	6.60	10.08

Table 3. Table 3: Results of cMDS-DML and LDMLR on the UMIST dataset, with different values of target neighbors K 𝐾 K and fixed n = 30 𝑛 30 n=30 .

$K$		cMDS-DML	LDMLR
4	MAE	0.3488	0.4291
	STD	$\pm 0.0684$	$\pm 0.0760$
	t(s)	2.88	10.26
5	MAE	0.3463	0.4676
	STD	$\pm 0.0671$	$\pm 0.0735$
	t(s)	3.10	10.39
6	MAE	0.3521	0.4782
	STD	$\pm 0.0689$	$\pm 0.0724$
	t(s)	3.32	11.92

Table 4. Table 4: Results of cMDS-DML and LDMLR on the UMIST dataset, with different values of dimension d 𝑑 d .

$d$		cMDS-DML	LDMLR
150	MAE	0.3463	0.4676
	STD	$\pm 0.0671$	$\pm 0.0735$
	t(s)	3.10	10.39
200	MAE	0.3518	0.4695
	STD	$\pm 0.0668$	$\pm 0.0730$
	t(s)	3.18	17.21
250	MAE	0.3545	0.4708
	STD	$\pm 0.0674$	$\pm 0.0742$
	t(s)	4.15	23.90

Table 5. Table 5: Results of cMDS-DML and LDMLR on the UMIST dataset, with different sizes of the training set n 𝑛 {n} .

$n$		cMDS-DML	LDMLR
30	MAE	0.3463	0.4676
	STD	$\pm 0.0671$	$\pm 0.0735$
	t(s)	3.10	10.39
60	MAE	0.1958	0.3103
	STD	$\pm 0.0478$	$\pm 0.0465$
	t(s)	4.94	39.13
90	MAE	0.1373	0.2264
	STD	$\pm 0.0329$	$\pm 0.0319$
	t(s)	5.91	79.92

Table 6. Table 6: Results of cMDS-DML and LDMLR on the FG-NET dataset, with different values of target neighbors K 𝐾 {K} .

$K$		cMDS-DML	LDMLR
4	MAE	0.7295	1.4179
	STD	$\pm 0.0694$	$\pm 0.1228$
	t(s)	2.70	21.59
5	MAE	0.7109	1.3482
	STD	$\pm 0.0657$	$\pm 0.0942$
	t(s)	3.11	18.37
6	MAE	0.7095	1.3278
	STD	$\pm 0.0682$	$\pm 0.1141$
	t(s)	3.86	25.43

Table 7. Table 7: Results of cMDS-DML and LDMLR on the FG-NET dataset, with different sizes of the training set n 𝑛 {n} .

$n$		cMDS-DML	LDMLR
40	MAE	0.7613	1.3634
	STD	$\pm 0.0845$	$\pm 0.1144$
	t(s)	2.34	11.13
50	MAE	0.7377	1.3482
	STD	$\pm 0.0657$	$\pm 0.0942$
	t(s)	4.56	18.37
75	MAE	0.7243	1.2551
	STD	$\pm 0.0809$	$\pm 0.1023$
	t(s)	29.97	40.04

Equations

$${\mathbf{x}}_1,\cdots ,{\mathbf{x}}_{i_1},$$

$${\mathbf{x}}_{i_1+1},\cdots ,{\mathbf{x}}_{i_2},$$

$$\cdots$$

$${\mathbf{x}}_{i_{m-1}+1},\cdots ,{\mathbf{x}}_{i_m},$$

$$d_A({\mathbf{x}}_i,{\mathbf{x}}_j)=\Vert {\mathbf{x}}_i-{\mathbf{x}}_j{\Vert }_A=\sqrt{({\mathbf{x}}_i-{\mathbf{x}}_j{)}^{T}A({\mathbf{x}}_i-{\mathbf{x}}_j)},$$

$$\begin{array}[]{ll}\hbox{min}_{A\in{\cal S}^{d}}&h(A)\\ \hbox{s.t.}&A\succeq 0,\end{array}$$

$$h(A)=-\sum _{i,j}{\omega }_{ij}{d}_A^2({\mathbf{x}}_i,{\mathbf{x}}_j)+\mu \sum _{{\eta }_{ij}=1}(d_A^2({\mathbf{x}}_i,{\mathbf{x}}_j)-d_{I_d}^2({\mathbf{x}}_i,{\mathbf{x}}_j){)}^2.$$

$$\eta_{ij}=\left\{\begin{array}[]{ll}1,&\hbox{ if }\mathbf{x}_{j}\hbox{ is the target neighbor of }\mathbf{x}_{i};\\ 0,&\text{otherwise}.\end{array}\right.$$

$$\omega_{ij}=\left\{\begin{array}[]{ll}(|r_{i}-r_{j}|+1)^{p}&\hbox{ if }r_{i}\neq r_{j};\\ 0&\hbox{ otherwise,}\end{array}\quad\textrm{where $p>0$.}\right.$$

$$A_{k+1}={\Pi }_{{\mathcal{S}}_{+}^d}(A_k-\nabla h(A_k)),$$

$$J:=I-\frac{1}{n}11^T\text{mbox}andB(D):=-\frac{1}{2}JDJ,$$

$$B(D)=[{\bf p}_{1},\ldots,{\bf p}_{s}]\left[\begin{array}[]{ccc}\lambda_{1}&&\\ &\ddots&\\ &&\lambda_{s}\end{array}\right]\left[\begin{array}[]{c}{\bf p}_{1}^{T}\\ \vdots\\ {\bf p}_{s}^{T}\end{array}\right],$$

$$[{\bf y}_{1},{\bf y}_{2},\ldots,{\bf y}_{n}]:=\left[\begin{array}[]{ccc}\sqrt{\lambda_{1}}&&\\ &\ddots&\\ &&\sqrt{\lambda_{s}}\end{array}\right]\left[\begin{array}[]{c}{\bf p}_{1}^{T}\\ \vdots\\ {\bf p}_{s}^{T}\end{array}\right]$$

$$\begin{array}{r}{d}_A({\mathbf{x}}_i,{\mathbf{x}}_j)=\sqrt{({\mathbf{x}}_i-{\mathbf{x}}_j{)}^T{L}^{T}L({\mathbf{x}}_i-{\mathbf{x}}_j)}=\Vert L({\mathbf{x}}_i-{\mathbf{x}}_j)\Vert :=d^L({\mathbf{x}}_i,{\mathbf{x}}_j).\end{array}$$

$$\Vert L{\mathbf{x}}_i-L{\mathbf{x}}_j\Vert >\Vert L{\mathbf{x}}_i-L{\mathbf{x}}_k\Vert ,if\; |r_i-r_j|>|r_i-r_k|,r_i\ne r_j,r_i\ne r_k,r_j\ne r_k.$$

$$d_A({\mathbf{x}}_i,{\mathbf{x}}_j)\approx d_{I_d}({\mathbf{x}}_i,{\mathbf{x}}_j),if{r}_i=r_{j}and{\mathbf{x}}_{j}is\; the\; target\; neighbor\; of{\mathbf{x}}_i.$$

$$D_{ij}=\left\{\begin{array}[]{ll}(|r_{i}-r_{j}|+\beta)^{2},&\text{if }r_{i}\neq r_{j};\\ 0,&\hbox{otherwise.}\end{array}\right.$$

$$\bar{\delta}_{ij}=\left\{\begin{array}[]{ll}|a_{i}-a_{j}|,&\text{if }i\neq j,\ i,j=1,\cdots,m;\\ 0,&\hbox{otherwise.}\end{array}\right.$$

$$\Vert {\mathbf{y}}_i-{\mathbf{y}}_j\Vert >\Vert {\mathbf{y}}_i-{\mathbf{y}}_k\Vert ,if\; |r_i-r_j|>|r_i-r_k|,r_i\ne r_j,r_i\ne r_k,r_j\ne r_k.$$

$${min}_{L\in \mathrm{I}{\mathrm{R}}^{s\times d},c\in \mathrm{I}\mathrm{R}}f(L,c):=\frac{1}{2}\sum _{i=1}^n\Vert L{\mathbf{x}}_i-c{\mathbf{y}}_i{\Vert }^2+\mu \sum _{{\eta }_{ij}=1}((d^L({\mathbf{x}}_i,{\mathbf{x}}_j){)}^2-d_{I_d}^2({\mathbf{x}}_i,{\mathbf{x}}_j){)}^2,$$

$$\Delta=(\delta_{ij}^{2}),\ \hbox{where}\ \ \delta_{ij}=\left\{\begin{array}[]{ll}\bar{\delta}_{ij}+\beta,&\text{if}\quad i\neq j,\ i,j=1,\cdots,m;\\ 0,&\hbox{otherwise.}\end{array}\right.$$

$$\Vert {\mathbf{y}}_i-{\mathbf{y}}_j\Vert =0,if{r}_i=r_j,$$

$$\mathrm{diag}(\Delta )=0andB(\Delta )=-\frac{1}{2}J\Delta J⪰0.$$

$$J\mathbf{missing}1=0,B(\Delta )J=B(\Delta ),JB(\Delta )=B(\Delta ).$$

$$B(\Delta )=B(\overline{\Delta })+2\beta B({\overline{\Delta }}^{\frac{1}{2}})+\frac{1}{2}{\beta }^{2}J.$$

$${\mathbf{x}}^{T}B(\overline{\Delta })\mathbf{x}+2\beta {\mathbf{x}}^{T}B({\overline{\Delta }}^{\frac{1}{2}})\mathbf{x}+\frac{1}{2}{\beta }^2{\mathbf{x}}^{T}J\mathbf{x}\ge 0.$$

$${\mathbf{x}}^T(B({\overline{\Delta }}^{\frac{1}{2}})-{\mu }_{0}I)\mathbf{x}\ge 0,\forall \mathbf{x}\in \mathrm{I}{\mathrm{R}}^m.$$

$${\mathbf{x}}^{T}B({\overline{\Delta }}^{\frac{1}{2}})\mathbf{x}-{\mu }_0{\mathbf{x}}^{T}J\mathbf{x}\ge 0,\forall \mathbf{x}\in \mathrm{I}{\mathrm{R}}^m.$$

$${\mathbf{x}}^{T}B(\overline{\Delta })\mathbf{x}+2\beta {\mathbf{x}}^{T}B({\overline{\Delta }}^{\frac{1}{2}})\mathbf{x}+\frac{1}{2}{\beta }^2{\mathbf{x}}^{T}J\mathbf{x}\ge \beta (2{\mu }_0+\frac{\beta }{2}){\mathbf{x}}^{T}J\mathbf{x}\ge 0,$$

$$f((L^k,c_k)+\gamma {\rho }^m{d}^k)-f(L^k,c_k)\le \sigma \gamma {\rho }^m\nabla f(L^k,c_k{)}^T{d}^k.$$

$$\begin{array}{rl}{\nabla }_{L}f(L,c)& =\sum _{i=1}^n(L{\mathbf{x}}_i{\mathbf{x}}_i^T-c{\mathbf{y}}_i{\mathbf{x}}_i^T)+4\mu \sum _{{\eta }_{ij}=1}L(X_{ij}{L}^{T}L{X}_{ij}-X_{ij}^2)\\ & =\sum _{i=1}^n(L{\mathbf{x}}_i{\mathbf{x}}_i^T-c{\mathbf{y}}_i{\mathbf{x}}_i^T)+4\mu \sum _{{\eta }_{ij}=1}(\Vert L({\mathbf{x}}_i-{\mathbf{x}}_j){\Vert }^2-\Vert {\mathbf{x}}_i-{\mathbf{x}}_j{\Vert }^2)L({\mathbf{x}}_i-{\mathbf{x}}_j)({\mathbf{x}}_i-{\mathbf{x}}_j{)}^T,\\ {\nabla }_{c}f(L,c)& =c\sum _{i=1}^n{\mathbf{y}}_i^T{\mathbf{y}}_i-\sum _{i=1}^n{\mathbf{y}}_i^{T}L{\mathbf{x}}_i.\end{array}$$

Full text

Ordinal Distance Metric Learning with MDS for Image Ranking

Panpan Yu

School of Mathematics and Statistics

Beijing Institute of Technology

Beijing, 100081, China

2120151335@bit.edu.cn

Qing-Na Li*

School of Mathematics and Statistics

Beijing Key Laboratory on MCAACI

Beijing Institute of Technology

Beijing, 100081, China

qnl@bit.edu.cn

Image ranking is to rank images based on some known ranked images. In this paper, we propose an improved linear ordinal distance metric learning approach based on the linear distance metric learning model in Li et al., (2015). By decomposing the distance metric $A$ as $L^{T}L$, the problem can be cast as looking for a linear map between two sets of points in different spaces, meanwhile maintaining some data structures. The ordinal relation of the labels can be maintained via classical multidimensional scaling, a popular tool for dimension reduction in statistics. A least squares fitting term is then introduced to the cost function, which can also maintain the local data structure. The resulting model is an unconstrained problem, and can better fit the data structure. Extensive numerical results demonstrate the improvement of the new approach over the linear distance metric learning model both in speed and ranking performance.

Keywords: Image ranking; distance metric learning; classical multidimensional scaling; optimization model.

††*Corresponding author

1 Introduction

Given a labeled image dataset (referred as the training set), image ranking is to find the most relevant images for a query image based on the training set. Different from binary classification and multi-classification, the labels of the training set in image ranking often have an order, for example, age. The two important and challenging aims for image ranking are as follows. The first aim is to find which class the query image belongs to, and the second is to find the most relevant images in the specific class. The first aim actually falls into ordinal regression in statistics, where different approaches have been proposed, see Gutierrez et al., (2016) for a survey on ordinal regression and Qiao, (2015), Wang et al., (2017) for the recent development. However, the second aim makes image ranking different from ordinal regression since the training images having the same label with query image need to be further ranked. Therefore, a direct extension of methods for ordinal regression is not appropriate for image ranking.

As for the second aim, to find the most relevant images, a natural way is to use Euclidean distance between images to measure their dissimilarities. However, as we will show later, in most cases, Euclidean distance is not appropriate for dissimilarity. A practical way is to learn a distance metric (denoted as $A$) to measure the distances between images. This is referred as distance metric learning (DML). Then for a query image, the most relevant images are those with smallest distances under metric $A$. Many DML methods have been developed for image classification and clustering tasks. For example, the SDP approach proposed by Xing et al., (2003), an online learning algorithm proposed by Shalev-Shwartz et al., (2004), a neighborhood component analysis (NCA) by Goldberger et al., (2004), and so on (Bar-hillel et al., (2003); Shen et al., (2010); Yang et al., (2007)). However, most of these methods didn’t assume the labels are ordered. Therefore, they can not be directly used for image ranking.

Recently, Li et al., (2015) firstly introduced ordinal DML for image ranking. By a carefully designed weighting factor based on ordinal labels, the ordinal relationship of the images is expected to be maintained. An alternating iterative update was proposed to solve the resulting nonlinear convex semidefinite programming model, which is basically a projected gradient algorithm.

On the other hand, multidimensional scaling (MDS) is an important method for dimension reduction, which has been widely used in signal processing, molecular conformation, psychometrics and social measurement. We refer to some monographs and surveys for more applications (Anjos and Lasserre, (2012); Borg and Groenen, (2005); Dattorro, (2008); Dokmanic et al., (2015); Liberti et al., (2014)). The idea of classical MDS (cMDS) is to embed the given objects into a low dimensional space based on a Euclidean distance matrix. Recently, there has been great progress in MDS, such as the semismooth Newton method for nearest Euclidean distance matrix problem (Qi, (2013); Qi and Yuan, (2014)), the inexact smoothing Newton method for nonmetric MDS (Li and Qi, (2017)), as well as the applications of MDS in nonlinear dimension reduction (Ding and Qi, (2016, 2017)), binary code learning (Dai et al., (2016)), and sensor network localization (Qi et al., (2013)).

Our Contributions Note that the distance metric $A$ in DML is positive semidefinite. We represent $A$ as $A=L^{T}L$, where $L$ is a rectangular matrix. The first contribution of our work is that we look for $L$ instead of $A$, which gets around of positive semidefinite constraint on $A$. As a result, our method does not need spectral decomposition in each iteration and thus has quite low computational complexity. Moreover, if $L$ has only a few rows, the obtained $A$ is low rank. This brings new insight on distance metric. Distances between images under $A$ are basically the Euclidean distance between new points in a new space. The second contribution is that we employ cMDS to get the ideal points in the new space, whose Euclidean distances keep the ordinal relations as the labels do. In other words, cMDS is a key step to achieve the goal of maintaining the ordinal relationship of the data. The third contribution is that we propose a new ordinal DML model, which concerns ordinal relations between images and maintains local data structure. Extensive experiments are conducted on two data sets: UMIST face dataset and FG-NET aging dataset. The results demonstrate the efficiency and improvement of the new approach over the linear DML model in Li et al., (2015) both in speed and ranking performance.

The organization of the paper is as follows. In Section 2, we give some preliminaries about DML model in Li et al., (2015) and cMDS. In Section 3, we propose our new approach, referred as cMDS-DML approach. In Section 4, we discuss the numerical algorithm to solve the resulting unconstrained problem. In Section 5 we report the numerical results to demonstrate the efficiency of the proposed model. Final conclusions are given in Section 6.

Notations. We use ${\cal S}^{n}$ to denote the space of symmetric matrices of $n\times n$, and ${\cal S}^{n}_{+}$ to denote the space of positive semidefinite matrices of $n\times n$, and $A\succeq 0$ means $A\in{\cal S}^{n}_{+}$. We use small bold letters to indicate vectors.

2 Preliminaries

In this section, we give a brief review on the linear DML model in Li et al., (2015) and then give some preliminaries on cMDS.

2.1 Problem Statement

Suppose $\mathcal{X}=\{(\mathbf{x}_{i},r_{i}):\ i=1,\cdots,n\}$ is the training set, where $\mathbf{x}_{i}\in{\rm I}\!{\rm R}^{d}$, $i=1,\cdots,n$, are the observed data, and $r_{i}\in{\rm I}\!{\rm R}$, $i=1,\cdots,n$, are the corresponding labels which have an order. $n$ is sample number of the training set. We need the following assumptions.

Assumption 1**.**

Suppose there are total $m$ different ordinal labels. Assume that the data in the training set are grouped as follows

[TABLE]

where $i_{m}=n$, and $a_{1},\cdots,a_{m}$ are distinct ordinal labels.

Assumption 2**.**

Suppose $\mathbf{x}_{1},\cdots,\mathbf{x}_{n}$ are zero-centralized, i.e., $\sum_{i=1}^{n}\mathbf{x}_{i}=0$.

To rank images, the distance metric learning approach uses the distance $d_{A}(\cdot,\cdot)$ defined by

[TABLE]

where $A\in{\cal S}^{d}$ is positive semidefinite. The goal is then to learn an appropriate $A$, such that the distances under metric $A$ between relevant images are small. Once $A$ is obtained, the most relevant images of a query image can be provided as those with smallest distances under $A$. To this end, one expects $A$ to have two properties. Firstly, ordinal information needs to be preserved under $A$, that is, for $\mathbf{x}_{i},\mathbf{x}_{j}$ with $r_{i}\neq r_{j}$, $d_{A}(\mathbf{x}_{i},\mathbf{x}_{j})$ is small when $|r_{i}-r_{j}|$ is small. Secondly, local geometry structure of the data needs to be maintained under $A$. That is, for $\mathbf{x}_{i},\mathbf{x}_{j}$ with $r_{i}=r_{j}$, $d_{A}(\mathbf{x}_{i},\mathbf{x}_{j})\approx d_{I_{d}}(\mathbf{x}_{i},\mathbf{x}_{j})$, where $I_{d}$ is the identity matrix of size $d$. See also Li et al., (2015).

2.2 Linear Distance Metric Learning for Ranking

As mentioned in the introduction, most DML approaches did not assume the labels are ordered. Li et al., (2015) firstly proposed a method named Linear Distance Metric Learning for Ranking (LDMLR), which dealt with ordinal labels. Below we briefly review the main idea of LDMLR.

To derive LDMLR, for each $\mathbf{x}_{i}$, we first specify $K$ nearest data points (under Euclidean distance) with the same label as its target neighbors. The LDMLR method is to learn a metric $A$ by solving the following nonlinear convex semidefinite programming problem:

[TABLE]

where

[TABLE]

Here $\mu>0$ is a tradeoff parameter. $\eta_{ij}\in\{0,1\}$ indicates whether $\mathbf{x}_{j}$ is one of $\mathbf{x}_{i}$’s target neighbors, i.e.,

[TABLE]

And $\omega_{ij}$ is a weighting factor defined as

[TABLE]

The first term of $h(A)$ can be viewed as a penalty term of the distance between two data points if they have different labels. The weighting factor $\omega_{ij}$ is used to adjust the importance of such distances. As we can see from the definition of $\omega_{ij}$, the larger $|r_{i}-r_{j}|$ is, the bigger $\omega_{ij}$ is. If $\mathbf{x}_{i}$ and $\mathbf{x}_{j}$ have the same label, we don’t want to maximize their distances, so $\omega_{ij}=0$ in this case. The second term of $h(A)$ is trying to maintain the local structure between the images with the same label. Model (1) is a convex model, and can be solved by state-of-art quadratic semidefinite programming packages, such as QSDP by Toh, (2007). In Li et al., (2015), the projected gradient method is applied to solve (1), i.e., the following update is used

[TABLE]

where $\Pi_{{\cal S}^{d}_{+}}(\cdot)$ denotes the projection onto ${\cal S}^{d}_{+}$.

In LDMLR, the ordinal relation of the images is maintained by introducing a weighting factor, which is calculated based on the ordinal labels. Furthermore, the local data structure can be kept by the second term in $h(A)$.

2.3 Classical Multidimensional Scaling (cMDS)

The aim of cMDS is to embed data in a lower dimensional space while preserving the distances between data. Given the coordinates of a set of points, namely $\{\mathbf{y}_{1},\ldots,\mathbf{y}_{n}\}$ with $\mathbf{y}_{i}\in{\rm I}\!{\rm R}^{s}$, it is straightforward to compute the pairwise Euclidean distances: $d_{ij}=\|\mathbf{y}_{i}-\mathbf{y}_{j}\|$, $i,j=1,\ldots,n$. The matrix $D=(d_{ij}^{2})$ is known as the (squared) Euclidean Distance Matrix (EDM) of those points. However, the inverse problem is more interesting and important. Suppose $D$ is given. The method of cMDS generates a set of coordinates that preserve the pairwise distances in $D$. We give a short description of cMDS below. Let

[TABLE]

where $I$ is the $n\times n$ identity matrix and ${\bf 1}$ is the (column) vector of all ones in ${\rm I}\!{\rm R}^{n}$. In literature, $J$ is known as the centralization matrix and $B$ is the double-centralized matrix of $D$ (also the Gram matrix of $D$ because $B$ is positive semidefinite). Suppose $B$ admits the spectral decomposition:

[TABLE]

where $\lambda_{1},\ldots,\lambda_{s}$ are positive eigenvalues of $B$ (the rest are zero) and ${\bf p}_{1},\ldots,{\bf p}_{s}$ are the corresponding orthonormal eigenvectors. Then the following coordinates ${\bf y}_{1},\ldots,{\bf y}_{n}$ obtained by

[TABLE]

preserve the known distances in the sense that $\|{\bf y}_{i}-{\bf y}_{j}\|=d_{ij}$ for all $i,j=1,\ldots,n$. This is the well known cMDS. We refer to Gower, (1985), Schoenberg, (1935), Torgerson, (1952), Young and Householder, (1938), Borg and Groenen, (2005), and Dattorro, (2008) for detailed description and generalizations of cMDS.

3 A New Approach for Ranking

In this section, we will motivate our new approach and discuss some related properties of EDM.

3.1 A New Approach

The idea of our approach is as follows. First, by decomposing $A=L^{T}L$, the problem reduces to looking for a linear map $L$ from the original space ${\rm I}\!{\rm R}^{d}$ to a new space, denoted as ${\rm I}\!{\rm R}^{s}$. The points $L\mathbf{x}_{i}$ in the new space are referred as the embedding points corresponding to $\mathbf{x}_{i}$, $i=1,\cdots,n$. Then we apply cMDS to get the estimations of those embedding points, denoted as $\{\mathbf{y}_{1},\cdots,\mathbf{y}_{n}\}$. Finally, $L$ is learned based on two sets of points $\{\mathbf{x}_{1},\cdots,\mathbf{x}_{n}\}$ and $\{\mathbf{y}_{1},\cdots,\mathbf{y}_{n}\}$. We detail our approach in the following three steps.

Step 1. Decompose $A=L^{T}L$

A natural way of learning a distance metric $A\in{\cal S}^{d}$ is to decompose $A$ as $A=L^{T}L$, where $L\in{\rm I}\!{\rm R}^{s\times d}$ is a rectangular matrix and $s$ is a prescribed dimension, where $s\leq d$. The decomposition has been used in several references, see for example Sugiyama, (2007), Weinberger and Saul, (2009), Xiang et al., (2008). Learning $L$ instead of $A$ brings us some advantages. Firstly, it allows us to get around of the positive semidefinite constraint $A\succeq 0$, resulting in an unconstrained model. Secondly, low rank structure of $A$ can be specified by choosing $s\ll d$. Note that given a query image, it is necessary to compute distances between the query image and every training image. The time complexity of computing distances should be kept as low as possible. With a low rank $A$, such complexity can be reduced from $O(d^{2})$ to $O(ds)$. Finally, it provides us insights on the Mahalanobis distance metric A. $L\in{\rm I}\!{\rm R}^{s\times d}$ is basically a linear map from ${\rm I}\!{\rm R}^{d}$ to ${\rm I}\!{\rm R}^{s}$. The distance between $\mathbf{x}_{i}$ and $\mathbf{x}_{j}$ under metric $A$ can be reformulated as

[TABLE]

In other words, the distance between $\mathbf{x}_{i}$ and $\mathbf{x}_{j}$ under metric $A$ is essentially the Euclidean distance of new points $L\mathbf{x}_{i}$ and $L\mathbf{x}_{j}$ in the space ${\rm I}\!{\rm R}^{s}$.

Recall that we denote the space where $\mathbf{x}_{i}$ lies in (i.e., ${\rm I}\!{\rm R}^{d}$) as the original space, the space where $L\mathbf{x}_{i}$ lies in (i.e., ${\rm I}\!{\rm R}^{s}$) as the new space, and $L\mathbf{x}_{i}$ is referred as the embedding point of $\mathbf{x}_{i}$. Now image ranking reduces to looking for a linear map, which maps $\mathbf{x}_{i}$ to a proper new space such that the following properties hold.

• (i)

The distances between embedding points can well reflect the corresponding ordinal labels. In other words, the Euclidean distances between embedding points with different labels should follow the order of their label differences, i.e.,

[TABLE]

• (ii)

Local data structure must be maintained. That is, the Euclidean distances between a point and its target neighbors with the same label in the original space need to be maintained as much as possible in the new space. That is,

[TABLE]

In the following, we apply cMDS to get the estimations $\{\mathbf{y}_{1},\cdots,\mathbf{y}_{n}\}$ of the embedding points in a new space, which enjoy property (i), then learn a linear mapping $L$ based on two sets of points $\{\mathbf{x}_{1},\cdots,\mathbf{x}_{n}\}$ and $\{\mathbf{y}_{1},\cdots,\mathbf{y}_{n}\}$.

Step 2. Apply cMDS

In order to apply cMDS to get the estimations of embedding points, an EDM is needed. Note that the points with the same label can be basically viewed as one point, and further inspired by the weighting factor defined in (3), we can construct an EDM based on the ordinal labels. A trivial choice is to define $D$ by $D_{ij}=(|r_{i}-r_{j}|^{2}),\ i,j=1,\cdots,n.$ However, from numerical point of view, we can further add a parameter $\beta$ to $|r_{i}-r_{j}|$ to allow more flexibility. This leads to the following form of $D$. Define $D\in{\cal S}^{n}$ as

[TABLE]

Under Assumption 1, let

[TABLE]

The following theorem shows that if $\beta$ is properly chosen, then $D$ is an EDM.

Theorem 1**.**

Let $\overline{\Delta}^{\frac{1}{2}}:=(\bar{\delta}_{ij})$ and $\mu_{0}$ is the smallest eigenvalue of $-\frac{1}{2}J\overline{\Delta}^{\frac{1}{2}}J$. If $\beta\geq-4\mu_{0}$, then $D$ defined by (8) is an EDM.

The proof is postponed in Section 3.2. If $D$ is not an EDM, we refer to Qi, (2013), Li and Qi, (2017) for more details. By applying cMDS to $D$, we can get the estimations $\{\mathbf{y}_{1},\cdots,\mathbf{y}_{n}\}$ of embedding points in the new space.

Remark 1**.**

For $\mathbf{x}_{i}$ and $\mathbf{x}_{j}$ with $r_{i}=r_{j}$, their estimations of embedding points $\mathbf{y}_{i},\mathbf{y}_{j}$ basically collapse to one point, since $\|\mathbf{y}_{i}-\mathbf{y}_{j}\|=D_{ij}=0$. For $\mathbf{x}_{i}$ and $\mathbf{x}_{j}$ with $r_{i}\neq r_{j}$, the Euclidean distance between their estimations $\mathbf{y}_{i},\ \mathbf{y}_{j}$ of embedding points is $\|\mathbf{y}_{i}-\mathbf{y}_{j}\|=D^{\frac{1}{2}}_{ij}=|r_{i}-r_{j}|+\beta$. Consequently, there is

[TABLE]

In other words, $\{\mathbf{y}_{1},\cdots,\mathbf{y}_{n}\}$ enjoy property $(i)$.

Step 3. Matching Two Sets of Points

The final step is to learn $L$ based on two sets of points $\{\mathbf{x}_{1},\cdots,\mathbf{x}_{n}\}$ and $\{\mathbf{y}_{1},\cdots,\mathbf{y}_{n}\}$ to make $L$ have properties (i) and (ii). To deal with property (i), we need to match $\{\mathbf{x}_{1},\cdots,\mathbf{x}_{n}\}$ and $\{\mathbf{y}_{1},\cdots,\mathbf{y}_{n}\}$ as much as possible since $\mathbf{y}_{1},\cdots,\mathbf{y}_{n}$ already satisfy property (i). A natural statistical way is to use a least squares fitting term. To tackle property (ii), we adopt the second term of $h(A)$ in (1), since it does a good job based on the numerical performance. Now we reach the following model

[TABLE]

where $\eta_{ij}$ is defined as in (2). To allow more flexibility, we also use a scaling variable $c\in{\rm I}\!{\rm R}$ in the fitting term.

Although (10) is a nonconvex model in $L$, the proposed approach enjoys the following good properties.

• •

By dealing with $L$ instead, the resulting model (10) is an unconstrained problem, which allows various numerical algorithms to solve. Further, we can emphasize the low rank structure of $A$ by restricting $L$ to be a short fat matrix, i.e., $s\ll d$.

• •

By applying cMDS, we take into account of the ordinal information of labels, which leads us a good estimation of embedding points.

• •

By matching $\{\mathbf{x}_{1},\cdots,\mathbf{x}_{n}\}$ with $\{\mathbf{y}_{1},\cdots,\mathbf{y}_{n}\}$ with the least squares fitting term, hopefully, the resulting embedding points will also keep property (i). Our numerical results actually verify this observation.

3.2 Proof of Theorem 1

Define $\Delta\in{\cal S}^{m}$ as

[TABLE]

Then we have the following lemma.

Lemma 1**.**

Let $D\in{\cal S}^{n}$ and $\Delta\in{\cal S}^{m}$ be defined as in (8) and (11). Let Assumption 2 hold. $D$ is an EDM if and only if $\Delta$ is an EDM.

Proof..

Suppose $D$ is an EDM generated by points $\{\mathbf{y}_{1},\cdots,\mathbf{y}_{n}\}$. By the definition of $D$, there is

[TABLE]

which implies that $\mathbf{y}_{i_{t-1}+1}=\cdots=\mathbf{y}_{i_{t}}$, $t=1,\cdots,m$. Let $\mathbf{y}_{i_{t-1}+1}=\cdots=\mathbf{y}_{i_{t}}:=\mathbf{z}_{t}$, $t=1,\cdots,m$. Obviously, $\Delta$ is an EDM generated by points $\{\mathbf{z}_{1},\cdots,\mathbf{z}_{m}\}$. Conversely, suppose that $\Delta$ is an EDM generated by points $\{\mathbf{z}_{1},\cdots,\mathbf{z}_{m}\}$. Let $\mathbf{y}_{i_{t-1}+1}=\cdots=\mathbf{y}_{i_{t}}=\mathbf{z}_{t}$, $t=1,\cdots,m$. One can show that $D$ is an EDM generated by $\{\mathbf{y}_{1},\cdots,\mathbf{y}_{n}\}$. The proof is finished. ∎

Next, we show that $\Delta$ is an EDM if $\beta$ is properly chosen.

Lemma 2**.**

Let $\overline{\Delta}^{\frac{1}{2}}:=(\bar{\delta}_{ij})$ and $\mu_{0}$ is the smallest eigenvalue of $-\frac{1}{2}J\overline{\Delta}^{\frac{1}{2}}J$. If $\beta\geq-4\mu_{0}$, then $\Delta$ defined by (11) is an EDM.

Proof..

It is well known (Schoenberg, (1935); Young and Householder, (1938)) that $\Delta$ is an EDM if and only if

[TABLE]

Also note that

[TABLE]

To prove that $\Delta$ is an EDM, we only need to show the positive semidefiniteness of $B(\Delta)$. Let $\overline{\Delta}=(\bar{\delta}_{ij}^{2})$. Note that

[TABLE]

It suffices to show if $\beta\geq-4\mu_{0}$, then for any $x\in{\rm I}\!{\rm R}^{m}$, there is

[TABLE]

Obviously, $\overline{\Delta}$ is an EDM. Consequently, for any $\mathbf{x}\in{\rm I}\!{\rm R}^{m}$, $\mathbf{x}^{T}B(\overline{\Delta})\mathbf{x}\geq 0$. Further, $B(\overline{\Delta}^{\frac{1}{2}})-\mu_{0}I\succeq 0$ implies that

[TABLE]

By substituting $\mathbf{x}$ by $J\mathbf{x}$ and noting equalities in (12), we have

[TABLE]

It gives that

[TABLE]

where the last inequality follows by the assumption $\beta\geq-4\mu_{0}$ as well as the positive semidefiniteness of $J$. The proof is finished. ∎

The proof of Lemma 2 is inspired by Theorem 1 in Cailliez, (1983). The difference is that $B(\overline{\Delta})$ is an EDM and $\beta$ is allowed to be negative in Lemma 2.

Proof of Theorem 1..

The result of Theorem 1 can be directly derived from Lemma 1 and Lemma 2. ∎

Remark 2**.**

Note that in cMDS, $\{\mathbf{y}_{1},\cdots,\mathbf{y}_{n}\}$ obtained from $D$ is not unique due to the eigenvalue decomposition of $B(D)$. However, ${\mathbf{y}_{1},\cdots,\mathbf{y}_{n}}$ are centralized, i.e., $\sum_{i=1}^{n}\mathbf{y}_{i}=0$. The computational cost for generating $\{\mathbf{y}_{1},\cdots,\mathbf{y}_{n}\}$ is $O(n^{3})$. If $n$ is large, the computational cost can be further reduced to $O(m^{3})$ by the following process, which is based on Lemma 1 and Lemma 2. It is easy to verify that ${\mathbf{y}_{1},\cdots,\mathbf{y}_{n}}$ generated by Algorithm 1 satisfy $\sum_{i=1}^{n}\mathbf{y}_{i}=0$, and the corresponding EDM is $D$ defined in (8).

4 Numerical Algorithm

Problem (10) is an unconstrained nonlinear problem, and can be solved by various algorithms. Here, we choose the traditional steepest descent method with the Armijo line search. The convergence result of the steepest descent method can be found in classical optimization books, e.g. Nocedal and Wright, (2006, P42). Algorithm 2 summarizes the details of our approach.

Implementations Let $X_{ij}:=(\mathbf{x}_{i}-\mathbf{x}_{j})(\mathbf{x}_{i}-\mathbf{x}_{j})^{T}$, the gradient $\nabla f(L,c)$ takes the following form

[TABLE]

Computational Complexity We compare the computational complexity (mainly in multiplication and division) of Algorithm 2 with that of LDMLR, and the details are summarized in Table 1, where steps with underline indicate the iterative steps. Note that if $n$ is large, S2 can be replaced by Algorithm 1 and the computational complexity for S2 can be further reduced from $O(n^{3})$ to $O(m^{3})$. For the iterative process S4-S6, the complexity for each iteration is $O(rnKd^{2})$, where $r$ is the maximum number for the line search loop. In contrast, for LDMLR, the computational complexity in each iteration is $O(\hbox{max}(n^{2}d^{2},nKd^{3}))$, which is higher than that of S3-S6 in Algorithm 2, no matter $n>d$ or $n<d$.

5 Numerical Results

In this section, we present some numerical results to verify the efficiency of the proposed model. To evaluate the performance of the model, we employ the following popular procedure to assess the image ranking model. For a given dataset, we divide it into the training set and the testing set. We first learn a distance metric based on the training set, then apply it to rank each image in the testing set. Denote by $\{\mathbf{m}_{i}\}_{i=1}^{N}$ the images in the testing set, here $N$ is the size of testing set. The estimated label $\hat{p}_{i}$ is obtained based on the distance in the new space. We employ the popular $k$-nearest neighbor regression to obtain $\hat{p}_{i}$, which is used in Li et al., (2015), Weinberger and Saul, (2009). The mean absolute error $MAE=1/N\sum_{i=1}^{N}|\hat{p}_{i}-p_{i}|$ is used as a measure to evaluate the performance. Here $p_{1},\cdots,p_{N}$ are the true labels of test data $\mathbf{m}_{1},\cdots,\mathbf{m}_{N}$.

We test the proposed method on the UMIST dataset (Graham and Allinson, (1998)) and FG-NET dataset (Lanitis, (2008)). We also compare our method with the method LDMLR in Li et al., (2015). For each test problem, we repeat each experiment 50 times and report the average results. The algorithm is implemented in Matlab R2016a and is run on a computer with Intel Core 2 Duo CPU E7500 2.93GHz, RAM 2GB.

5.1 Experiments on the UMIST image dataset

The UMIST face dataset is a multiview dataset which consists of 575 images of 20 people, each covers a wide range of poses from profile to frontal views. Fig. 1 shows some examples from the UMIST dataset.

Based on the query man wearing glasses, we can label the dataset in the following way: man wearing glasses is regarded as completely relevant, which is labeled as 2 in our experiment; man not wearing glasses or woman wearing glasses is regarded as partially relevant, which is labeled as 1; woman not wearing glasses is regarded as irrelevant, which is labeled as 0. Thus, there are 225, 239 and 111 images in the three categories, respectively. The dimension of original data is 10304.

In this experiment, for LDMLR, we set iteration number $T_{\hbox{max}}=30$ and the tradeoff parameter $\mu=10^{3}$ according to Li et al., (2015). For our method, we set parameters $\mu=10^{-10}$, $\gamma=10^{-9}$, $\rho=0.5$, $\sigma=0.05$, the maximum number for line search loop is $r=20$. To get an EDM $D$ in $(\ref{D})$, we set parameter $\beta=1$ ($\mu_{0}=0$ in this situation). To apply our algorithm, we first use PCA to reduce dimension as done in Li et al., (2015). When using PCA, we center the data but don’t scale the data. The final dimension is $150$, i.e., $d=150$.

Role of the Embedding Dimension $s$ and Distance Metric To see the role of the Embedding dimension $s$ and distance metric, we do the following test. We randomly select 10 images from each label for training and use the rest for testing. The images in the training set are grouped as follows. The training data $\mathbf{x}_{1},\cdots,\mathbf{x}_{10}$ are of label $1$, $\mathbf{x}_{11},\cdots,\mathbf{x}_{20}$ are of label $2$, and $\mathbf{x}_{21},\cdots,\mathbf{x}_{30}$ are of label $3$. Then there are $n=30$ training data in total. We fix the number of target neighbors as $K=5$.

To choose a proper embedding dimension, we tried several values for $s$, i.e., $s=2,3,5,8,10$. The preliminary results are reported in Table 2. Since $n=30$ is not so big, we directly apply cMDS to $D$ in S2 of Algorithm 2. The observation is that $s=3$ and $s=5$ are the best in terms of MSE. Taking visualization into account, we choose $s=3$ in our following test.

Then we compute the Euclidean distance between the training data $\mathbf{x}_{i},\ \mathbf{x}_{j},\ i,j=1,\cdots,30$. Fig. 2 shows $\|\mathbf{x}_{i}-\mathbf{x}_{1}\|$, the Euclidean distance between $\mathbf{x}_{i}$ and the first data $\mathbf{x}_{1}$, $i=1,\cdots,n$. It is observed that the distance between $\mathbf{x}_{1}$ and $\mathbf{x}_{12}$ is less than the distance between $\mathbf{x}_{1}$ and $\mathbf{x}_{8}$. Moreover, the distance between $\mathbf{x}_{1}$ and $\mathbf{x}_{16}$ is bigger than the distance between $\mathbf{x}_{1}$ and $\mathbf{x}_{22}$. It implies that the Euclidean distances between the original images can not be used for ranking. With embedding dimension $s=3$, we apply our method to learn $L$. After learning $L$, the embedding points of the training data in the three dimensional space can be found, i.e., $L\mathbf{x}_{i}$, $i=1,\cdots,30$. Fig. 3 plots the embedding points. As we can see, points highly cluster together with the same label. However, the distances between points with different labels can not be clearly seen from Fig. 3. We use the learned $L$ to measure the distances between the training data. Fig. 4 illustrates the distances between $\mathbf{x}_{i}$ and $\mathbf{x}_{1}$ under $L$, i.e., $\|L\mathbf{x}_{i}-L\mathbf{x}_{1}\|$, $i=1,\cdots,n$. Comparing Fig. 4 with Fig. 2, we can see that the data is much better layered with the $L$ distance than with the Euclidean distance. Hence the proposed model does preserve the ordinal relationship.

Comparison with LDMLR Now we compare with LDMLR in Li et al., (2015). First, we randomly select 10 images from each distinct label as the training data and use the rest for testing. Different values of $K$ are chosen to investigate the performance. Table 3 gives the results including MAE, STD (standard deviation), and CPU time in seconds. We can see in all cases, cMDS-DML uses much less time than LDMLR, which is not surprising since our method has lower computational complexity. In terms of MAE, cMDS-DML also outperforms LDMLR.

Next, to evaluate the influence of dimension on the performance of our method, we increase dimension $d$ while fixing the size of the training set $n=30$ and the number of target neighbors $K=5$. Table 4 lists the ranking results. It can be seen that as $d$ increases, the resulting MAE of both algorithms is not sensitive to $d$. As for computing time, as $d$ increases, LDMLR obviously costs more time while CUP time for our method is fairly stable. It’s reasonable since the computational complexity of our method is proportional to $d^{2}$ while that of LDMLR is $d^{3}$.

Finally, we increase the size of the training set $n$ with fixed dimension $d=150$ and the number of target neighbors $K=5$. We randomly select $n/3$ images from each distinct label for training and report results in Table 5. As $n$ increases, the performance of both methods becomes better, which is reasonable. cMDS-DML achieves higher ranking performance than LDMLR. In particular, cMDS-DML achieves $25.94\%$, $36.90\%$, $39.36\%$ improvement in MAE ($|$MAE(LDMLR)-MAE(cMDS-DML)$|$/MAE(LDMLR)) over LDMLR, respectively. Moreover, cMDS-DML is also faster than LDMLR.

5.2 Experiments on the FG-NET dataset

In this experiment, we test our algorithm on the FG-NET dataset which is labeled by age. The FG-NET dataset contains 1002 face images. There are 82 subjects in total with the age ranges from 1 to 69. Fig. 5 shows some examples from the FG-NET dataset. To get better performance of LDMLR, we set the iteration number $T_{\hbox{max}}=50$ and the tradeoff parameter $\mu=10^{3}$. For our method, $\mu=10^{-10}$, $\gamma=10$, $\rho=0.5$, $\sigma=0.05$, the embedding dimension $s=3$ and the maximum number for line search loop is $r=20$.

We pick up subjects with age 1, 5, 9, 15, 19 and relabel them as 1, 2, 3, 4, 5. There are 27, 40, 25, 30, 23 images in the five categories, respectively. The original dimension of images is 136. As in subsection $5.1$, we preprocess the data by PCA to reduce dimension to $80$. We randomly select 8 images from each distinct label for training and set $K=5$. Fig. 6 plots the embedding data points of the training data points in the three dimensional space, i.e., $L\mathbf{x}_{i}$, $i=1,\cdots,40$. As we can see, points almost cluster together with the same label.

Next, we randomly select 10 images from each distinct label for training and use the rest for testing. That is, the size of the training set is $n=50$. We also set $\beta=1$ in $(\ref{D})$. We set different values for target neighbors to investigate the performance. Table 6 lists the experimental results. In the three cases, cMDS-DML achieves $48.55\%$, $47.27\%$, $46.57\%$ improvement over LDMLR, respectively.

Finally, we fix the value of target neighbors $K=5$. We randomly select $n/5$ images from each distinct label for training. The size of the training set is chosen as $n=40,\ 50,\ 75$. See Table 7 for the results, which again verify the efficiency of the proposed model.

Overall speaking, our numerical results show that cMDS-DML outperforms LDMLR significantly both in ranking performance and CPU time.

6 Conclusions

In this paper, we proposed a so-called cMDS-DML approach for image ranking, which unifies the idea of classical multidimensional scaling and distance metric learning. The algorithm enjoys low computational complexity, compared with LDMLR in Li et al., (2015). Numerical results verified the efficiency of the new approach and the improvement over LDMLR.