SoDeep: a Sorting Deep net to learn ranking loss surrogates

TL;DR

SoDeep introduces a deep learning approach to approximate non-differentiable ranking metrics, enabling their use as differentiable objectives across various ranking tasks with competitive results.

Contribution

The paper presents a novel deep architecture, SoDeep, that learns to approximate sorting operations, facilitating the optimization of non-differentiable ranking metrics.

Findings

Achieves competitive results in text-image retrieval

Effective in multi-label image classification

Validates flexibility across ranking tasks

Abstract

Several tasks in machine learning are evaluated using non-differentiable metrics such as mean average precision or Spearman correlation. However, their non-differentiability prevents from using them as objective functions in a learning framework. Surrogate and relaxation methods exist but tend to be specific to a given metric. In the present work, we introduce a new method to learn approximations of such non-differentiable objective functions. Our approach is based on a deep architecture that approximates the sorting of arbitrary sets of scores. It is trained virtually for free using synthetic data. This sorting deep (SoDeep) net can then be combined in a plug-and-play manner with existing deep architectures. We demonstrate the interest of our approach in three different tasks that require ranking: Cross-modal text-image retrieval, multi-label image classification and visual memorability ranking. Our approach yields very competitive results on these three tasks, which validates the merit and the flexibility of SoDeep as a proxy for sorting operation in ranking-based losses.

Tables

Table 1. Table 1 : Performance of the sorters on synthetic data . Ranking performance of the sorter on the synthetic dataset. Among the learned sorters the LSTM one is the most efficient.

\rowcolorgray!40 Sorter model	L1 loss
Handcrafted sorter	0.0350
CNN sorter	0.0120
LSTM sorter loss	0.0033

Table 2. Table 2 : Media Memorability prediction results . Our proposed loss function and architecture outperform the state-of-the-art system [ 17 ] by 0.6 pt.

\rowcolorgray!40 Single model	Spear. cor. test
Baseline [6]	46.0
Image only [17]	48.8
R34 + MSE loss	44.2
R34 + SoDeep loss	46.6
Sem-Emb + MSE loss	48.6
Sem-Emb + SoDeep loss	49.4

Table 3. Table 3 : Cross-modal retrieval results on MS-COCO . Using the proposed rank based loss function outperforms the hard negative triplet margin loss, achieving state-of-the-art results on the caption retrieval task.

\rowcolorgray!40		caption retrieval					image retrieval
\rowcolorgray!40 model		R@1	R@5	R@10	Med. r		R@1	R@5	R@10	Med. r
Emb. network [37]		54.9	84.0	92.2	-		43.3	76.4	87.5	-
DSVE-Loc [10]		69.8	91.9	96.6	1		55.9	86.9	94.0	1
GXN (i2t+t2i) [16]		68.5	-	97.9	1		56.6	-	94.5	1
DSVE-Loc + SoDeep loss		71.5	92.8	97.1	1		56.2	87.0	94.3	1

Table 4. Table 4 : Object recognition results . Model marked by (*) are obtained with code available online: https://github.com/durandtibo/wildcat.pytorch

\rowcolorgray!40 Loss	mAP
VGG 16 [35]	89.3
WILDCAT [9]	95.0
WILDCAT*	93.2
WILDCAT* + SoDeep loss	94.0

Equations

$$\underset{{\mathbf{\Theta }}_B}{\min }\sum _{n=1}^N{\Vert \mathbf{rk}({\mathbf{y}}^{(n)})-f_{{\mathbf{\Theta }}_B}({\mathbf{y}}^{(n)})\Vert }_1.$$

$$r_s=1-\frac{6\Vert \mathbf{rk}(\mathbf{y})-\mathbf{rk}({\mathbf{y}}^{\prime }){\Vert }_2^2}{d(d^2-1)}.$$

$$\underset{{\mathbf{\Theta }}_A}{\min }\sum _{n=1}^N{\Vert \mathbf{rk}({\mathbf{y}}^{(n)})-\mathbf{rk}({\mathbf{y}}^{\ast (n)})\Vert }_2^2,$$

$${\mathcal{L}}_{SPR}({\mathbf{\Theta }}_A,\mathcal{B})=\sum _{n=1}^N{\Vert f_{{\mathbf{\Theta }}_B}(\mathbf{y}({\mathbf{\Theta }}_A{)}^{(n)})-\mathbf{rk}({\mathbf{y}}^{\ast (n)})\Vert }_2^2.$$

$$AP({\mathbf{y}}_c,{\mathbf{y}}_c^{\ast })=\frac{1}{\mathrm{rel}}\sum _{j:{\mathbf{y}}_c^{\ast }(j)=1}Prec(j),$$

$$Prec(j)=\frac{|\{s\in \mathcal{S}:{\mathbf{y}}_c^{\ast }(s)=1\}|}{\mathbf{rk}({\mathbf{y}}_c{)}_j},$$

$${\mathcal{L}}_{mAP}({\mathbf{\Theta }}_A,\mathcal{B})=\sum _{c=1}^C⟨f_{{\mathbf{\Theta }}_B}({\mathbf{y}}_c),{\mathbf{y}}_c^{\ast }⟩.$$

$$R@K(\mathbf{Y})=\frac{1}{d}\sum _{i=1}^d\{\begin{array}{ll}1,& \text{if }\mathbf{rk}(\mathbf{Y}[i]{)}_p<K\\ 0,& \text{otherwise,}\end{array}$$

$$\mathrm{loss}(\mathbf{Y}[i],p,c)=\max\big{\{}0,\alpha+f_{\boldsymbol{\Theta}_{B}}(\mathbf{Y}[i])_{p}-f_{\boldsymbol{\Theta}_{B}}(\mathbf{Y}[i])_{c}\big{\}},$$

$${\mathcal{L}}_{REC}({\mathbf{\Theta }}_A,\mathcal{B})=\frac{1}{d}\sum _{i\in \mathcal{B}}\underset{c\ne p,c\ne i}{\max }\mathrm{loss}(\mathbf{Y}[i],p,c).$$

$${\sigma }_{comp}(a,b)=\frac{1}{1+e^{-\lambda (b-a)}}.$$

$$f_h(\mathbf{y},i)=\sum _{j:j\ne i}{\sigma }_{comp}({\mathbf{y}}_i,{\mathbf{y}}_j).$$

Full text

In the present work, we introduce a new method to learn approximations of such non-differentiable objective functions. Our approach is based on a deep architecture that approximates the sorting of arbitrary sets of scores. It is trained virtually for free using synthetic data. This sorting deep (SoDeep) net can then be combined in a plug-and-play manner with existing deep architectures. We demonstrate the interest of our approach in three different tasks that require ranking: Cross-modal text-image retrieval, multi-label image classification and visual memorability ranking. Our approach yields very competitive results on these three tasks, which validates the merit and the flexibility of SoDeep as a proxy for sorting operation in ranking-based losses.

1 Introduction

Deep learning approaches have gained enormous research interest for many Computer Vision tasks in the recent years. Deep convolutional networks are now commonly used to learn state-of-the-art models for visual recognition, including image classification [26, 18, 35] and visual semantic embedding [25, 22, 37]. One of the strengths of these deep approaches is the ability to train them in an end-to-end manner removing the need for handcrafted features [29]. In such a paradigm, the network starts with the raw inputs, and handles feature extraction (low level and high-level features) and prediction internally. The main requirement is to define a trainable scheme. For deep architectures, stochastic gradient descent with back-propagation is usually performed to minimize an objective function. This loss function depends on the target task but has to be at least differentiable.

Machine learning tasks are often evaluated and compared using metrics which differ from the objective function used during training. The choice of an evaluation metric is intimately related to the definition of the task at hand, even sometimes to the benchmark itself. For example, accuracy seems to be the natural choice to evaluate classification methods, whereas the choice of the objective function is also influenced by the mathematical properties that allow a proper optimization of the model. For classification, one would typically choose the cross entropy loss – a differentiable function – over the non-differentiable accuracy. Ideally, the objective function used during training would be identical to the evaluation metric. However, standard evaluation metrics are often not suitable as training objectives for lack of differentiability to start with. This results in the use of surrogate loss functions that are better behaved (smooth, possibly convex). Unfortunately, coming up with good surrogate functions is not an easy task.

In this paper, we focus on the non-differentiability of the evaluation metrics used in ranking-based tasks such as recall, mean average precision and Spearman correlation. Departing from prior art on building surrogates losses for such tasks, we adopt a simple, yet effective, learning approach: Our main idea is to approximate the non-differentiable part of such ranking-based metrics by an all-purpose learnable deep neural network. In effect, this architecture is designed and trained to mimic sorting operations. We call it SoDeep. SoDeep can be added in a plug-and-play manner on top of any deep network trained for tasks whose final evaluation metric is rank-based, hence not differentiable. The resulting combined architecture is end-to-end learnable with a loss that relates closely to the final metric.

Our contributions are as follows:

• •

We propose a deep neural net that acts as a differentiable proxy for ranking, allowing one to rewrite different evaluation metrics as functions of this sorter, hence making them differentiable and suitable as training loss.

• •

We explore two types of architectures for this trainable sorting function: convolutional and recurrent.

• •

We combine the proposed differentiable sorting module with standard deep CNNs, train them end-to-end on three challenging tasks, and demonstrate the merit of this novel approach through extensive evaluations of the resulting models.

The rest of the paper is organized as follows. We discuss in Section 2 the related works on direct and indirect optimization of ranking-based metrics, and position our work accordingly. Section 3 is dedicated to the presentation of our approach. We show in particular how a “universal” sorting proxy suffices to tackle standard rank-based metrics, and present different architectures to this end. More details on the system and its training are reported in Section 4, along with various experiments. We first establish new state-of-the-art performance on cross-modal retrieval, then we show the benefits of our learned loss function compared to standard methods on memorability prediction and multi-label image classification.

2 Related works

Many data processing systems rely on sorting operations at some stage of their pipeline. It is the case also in machine learning, where handling such non-differentiable, non-local operations can be a real challenge [32]. For example, retrieval systems require to rank a set of database items according to their relevance to a query. For sake of training, simple loss functions that are decomposable over each training sample have been proposed as for instance in [19] for the area under the ROC curve. Recently, some more complex non-decomposable losses (such as the Average Precision (AP), Spearman coefficient, and normalized discounted cumulative gain (nDCG) [3]) that present hard computational challenges have been proposed [31].

Mean average precision optimization

Our work shares the high level goal of using ranking metrics as training objective function with many works before us. Several works studied the problem of optimizing average precision with support vector machines [21, 40] and other works extended these approaches to neural networks [1, 31, 8]. To learn rank, the seminal work [21] relies on a structured hinge upper bound to the loss. Further works reduce the computational complexity [31] or rely on asymptotic methods [36]. The focus of these works is mainly on the relaxation of the mean average precision, while our focus is on learning a surrogate for the ranking operation itself such that it can be combined with multiple ranking metrics. In contrast to most ranking-based techniques, which have to face the high computational complexity of the loss augmented inference [21, 36, 31], we propose a fast, generic, deep sorting architecture that can be used in gradient-based training for rank-based tasks.

Application of ranking based metrics

Ranking is commonly used in evaluation metrics. On retrieval tasks such as cross-modal retrieval [25, 22, 15, 12, 30], recall is the standard evaluation. Image classification [11, 9] and object recognition are evaluated with mean average precision in the multi-label case. Ordinal regression [5] is evaluated using Spearman correlation.

Existing surrogate functions

Multiple surrogates for ranking exist. Using metric learning to do retrieval is one of them. This popular approach avoids the use of the ranking function altogether. Instead, pairwise [39], triplet-wise [38, 4] and list-wise [13, 2] losses are used to optimize distances in a latent space. The cross-entropy loss is typically used for multi-label and multi-class classification tasks.

3 SoDeep approach

Rank-based metrics such as recall, Spearman correlation and mean average precision can be expressed as a function of the rank of the output scores. The computation of the rank being the only non-differentiable part of these metrics, we propose to learn a surrogate network that approximates directly this sorting operation.

3.1 Learning a sorting proxy

Let $\mathbf{y}\in\mathbb{R}^{d}$ be a vector of $d$ real values and $\mathbf{rk}$ the ranking function so that $\mathbf{rk}(\mathbf{y})\in\{1\cdots d\}^{d}$ is the vector containing the rank for each variable in $\mathbf{y}$, i.e.  $\mathbf{rk}(\mathbf{y})_{i}$ is the rank of $\mathbf{y}_{i}$ among the $\mathbf{y}_{j}$’s. We want to design a deep architecture $f_{\boldsymbol{\Theta}_{B}}$ that is able to mimic this sorting operator. The training procedure of this DNN is summarized in Fig. 2. The aim is to learn its parameters, $\boldsymbol{\Theta}_{B}$, so that the output of the network is as close as possible to the output of the exact sorting.

Before discussing possible architectures, let’s consider the training of this network, independent of its future use. We first generate a training set by randomly sampling $N$ input vectors $\mathbf{y}^{(n)}$ and we compute through exact sorting the associated ground-truth rank vectors $\mathbf{r}^{(n)}=\mathbf{rk}(\mathbf{y}^{(n)})$. We then classically learn the DNN $f_{\boldsymbol{\Theta}_{B}}$ by minimizing a $L_{1}$ loss between the predicted ranking vector $\mathbf{\hat{r}}=f_{\boldsymbol{\Theta}_{B}}(\mathbf{y})$ and the ground-truth rank $\mathbf{r}$ over the training set:

[TABLE]

We explore in the following different network architectures and we explain how the training data is generated.

3.1.1 Sorter architectures

We investigate two types of architectures for our differentiable sorter $f_{\boldsymbol{\Theta}_{B}}$. One is a recurrent network and the other one a convolutional network, each capturing interesting aspects of standard sorting algorithms:

• •

The recurrent architecture in Fig. 3(a) consists of a bi-directional LSTM [34] followed by a linear projection. The bi-directional recurrent network creates a connection between the output of the network and every input, which is critical for ranking computation. Knowledge about the whole sequence is needed to compute the true rank of any element.

• •

The convolutional architecture in Fig. 3(b) consists of 8 convolutional blocks, each of these blocks being a one-dimensional convolution followed by a batch normalization layer [20] and a ReLU activation function. The sizes of the convolutional filters are chosen such that the output of the network contains as many channels as the length of the input sequence. Convolutions are used for their local property: indeed, sorting algorithms such as bubble sort [14] only rely on a sequence of local operations. The intuition is that a deep enough convolutional network, with its cascaded local operations, should be able to mimic recursive sorting algorithms and thus to provide an efficient approximation of ranks.

We will further discuss the interest of both types of SoDeep block architectures in the experiments.

3.1.2 Training data

SoDeep module can be easily (pre)trained with supervision on synthetic data. Indeed, while being non-differentiable, the ranking function $\mathbf{rk}$ can be computed with classic sorting algorithms. The training data consists of vectors of randomly generated scalars, associated with their ground-truth rank vectors. In our experiments, the numbers are sampled from different types of distributions:

• •

Uniform distribution over $[-1,1]$;

• •

Normal distribution with $\mu=0$ and $\sigma=1$;

• •

Sequence of evenly spaced numbers in a uniformly drawn random sub-range of $[-1,1]$;

• •

Random mixtures of the previous distributions.

While the differentiable sorter can be trained ahead of time on a variety of input distributions, as explained above, there might be a shift with the actual score distribution that the main network $f_{\boldsymbol{\Theta}_{A}}$ will output for the task at hand. This shift can reduce naturally during training, or an alignment can be explicitly enforced. For example, $f_{\boldsymbol{\Theta}_{A}}$ can be designed to output data in the interval used to learn the sorter, with the help of bounded functions such as cosine similarity.

3.2 Using SoDeep for training with rank-based loss

Rank-based metrics are used for evaluating and comparing learned models in a number of tasks. Recall is a standard metric for image and information retrieval, mean Average Prediction (mAP) for classification and recognition, and Spearman correlation for ordinal prediction. This type of rank-based metrics are non-differentiable because they require to transition from the continuous domain (score) toward the discrete domain (rank).

As presented in Fig 1, we propose to insert a pre-trained SoDeep proxy block $f_{\boldsymbol{\Theta}_{B}}$ between the deep scoring function $f_{\boldsymbol{\Theta}_{A}}$ and the chosen rank-based loss. We show in the following how mAP, Spearman correlation and recall can be expressed as functions of the rank and combined with SoDeep accordingly.

In the following we assume a training set of annotated pairs $\{(\mathbf{x}_{i},y_{i}^{*})\}_{i=1}^{M}$ for the task at hand. A group $\mathcal{B}$ of $d$ training examples among them yields a prediction vector $\mathbf{y}(\boldsymbol{\Theta}_{A})=[f_{\boldsymbol{\Theta}_{A}}(\mathbf{x}_{i})]_{i\in\mathcal{B}}$ and an associated ground-truth score vector $\mathbf{y}^{*}=[y_{i}^{*}]_{i\in\mathcal{B}}$ (Fig. 1).

3.2.1 Spearman correlation

For two vectors $\mathbf{y}$ and $\mathbf{y}^{\prime}$ of size $d$, corresponding to two sets of $d$ observations, the Spearman correlation [7] is defined as:

[TABLE]

Maximizing w.r.t. parameters $\boldsymbol{\Theta}_{A}$ the sum of Spearman correlations (2) between ground truth and predicted score vectors over $N$ subsets of training examples amounts to solving the minimization problem:

[TABLE]

with the loss not being differentiable.

Using now our differentiable proxy instead of the rank function, we can define the new Spearman loss for a group $\mathcal{B}$:

[TABLE]

Training will typically minimize it over a large set of groups. Note that here the optimization is done over $\boldsymbol{\Theta}_{A}$, knowing that SoDepp block $f_{\boldsymbol{\Theta}_{B}}$ has been trained independently on specific synthetic training data. Optionally, the block can be fine-tuned along the way, hence minimizing w.r.t. $\boldsymbol{\Theta}_{B}$ as well.

3.2.2 Mean Average Precision (mAP)

Multilabel image classification is often evaluated using mAP, a metric from information retrieval. To define it, each of the $C$ classes is considered as a query over the $d$ elements of the datasets. For class $c$, denoting $\mathbf{y}_{c}^{*}$ the $d$-dimensional ground-truth binary vector and $\mathbf{y}_{c}$ the vector of scores for this class, the average precision (AP) for the class is defined as [40] :

[TABLE]

where $\mathrm{rel}=|j:\mathbf{y}^{*}_{c}(j)=1|$ is the number of positive items for class $c$ and precision for element $j$ is defined as:

[TABLE]

with $\mathcal{S}$ the set of indices of the elements of $\mathbf{y}_{c}$ larger than $\mathbf{y}_{c}(j)$.

Minimizing $\mathbf{rk}(\mathbf{y})_{j}$ for all $j$ from class $c$ (i.e., those verifying $\mathbf{y}^{*}_{c}(j)=1$) will be used as a surrogate of the maximization of the AP over predictor’s parameters $\boldsymbol{\Theta}_{A}$.

The mAP is obtained by averaging AP over the $C$ classes. Replacing the rank function by its differentiable proxy, the proposed mAP-based loss reads:

[TABLE]

3.2.3 Recall at $K$

Recall at rank $k$ is often used to evaluate retrieval tasks. In the following we assume a training set $\{\mathbf{x}_{i}\}_{i=1}^{M}$ for the task at hand. A group $\mathcal{B}$ of $d$ training examples among them yields a $d\times d$ prediction matrix $\mathbf{Y}(\boldsymbol{\Theta}_{A})=[f_{\boldsymbol{\Theta}_{A}}(\mathbf{x}_{i})]_{i\in\mathcal{B}}$ representing the scores of all pairwise combinations of training examples in $\mathcal{B}$. In other words, the $i$-th column of this matrix, $\mathbf{Y}[i]=f_{\boldsymbol{\Theta}_{A}}(\mathbf{x}_{i})$, provides the relevance of other vectors in the group w.r.t. to query $\mathbf{x}_{i}$.

This matrix being given, recall at $K$ is defined as:

[TABLE]

with $p$ the index of the unique positive entry in $\mathbf{Y}[i]$, a single relevant item being assumed for query $\mathbf{x}_{i}$.

Once again, our sorter enables a differentiable implementation of this measure. However, we could not obtain conclusive results yet, possibly due to the batch size limiting the range of the summation. We found, however, an alternative way to leverage our sorting network. It is based on the use of the “triplet loss”, a popular surrogate for recall. We propose to apply this loss on ranks instead of similarity scores, making it only dependent on the ordering of the retrieved elements. The triplet loss on the rank can be expressed as follows:

[TABLE]

where $p$ is defined as above (the positive example in the triplet, given anchor query $\mathbf{x}_{i}$) and $c$ is the index of a negative (irrelevant) example for this query. The goal is to minimize the rank of the positive pair with score $\mathbf{Y}[i]_{p}$ such that its rank is lower than the rank of the negative pair with score $\mathbf{Y}[i]_{c}$ by a margin of $\alpha$.

The complete loss is then expressed over all the elements of $\mathcal{B}$ in its hard negative version as:

[TABLE]

In equations (2), (5) and (8), the metrics are expressed in function of the non-differentiable rank function $\mathbf{rk}$. Leveraging our differentiable surrogate allows us to design a differentiable loss function for each of these metrics, respectively (4), (7) and (10).

4 Experiments

We present in this section several experiments to evaluate our approach. We first detail the way we train our differentiable sorter deep block using only synthetic data. We also present a comparison between the different models based on CNNs and on LSTM recurrent nets and with our baseline inspired from pairwise comparisons. We then evaluate the SoDeep combined with deep scoring functions $f_{\boldsymbol{\Theta}_{B}}$. The loss functions expressed in (4), (7) and (10) are applied to three different tasks: memorability prediction, cross-modal retrieval, and object recognition.

4.1 SoDeep Training and Analysis

4.1.1 Training

The proposed SoDeep models based on BI-LSTM and CNNs are trained on synthetic pairs of scores and ranks generated on the fly according to the distributions defined in Section 3.1.2.

For convenience we call an epoch as going through 100 000 pairs. The training is done using the Adam optimizer [24] with a learning rate of $0.001$ which is halved every 100 epochs. Mini-batches of size 512 are used. The model is trained until the loss values stop decreasing and are stable.

4.1.2 A handcrafted sorting baseline

We add to our trainable SoDeep blocks a baseline that does not require any training.

Inspired by the representation of the ranking problem as a matrix of pairwise ordering in [40], we build a handcrafted differentiable sorter $f_{h}$ using pairwise comparisons.

A sigmoid function parametrized with $\lambda$ scalar is used as a binary comparison function between two scalars $a$ and $b$ as:

[TABLE]

Indeed, if $a$ and $b$ are separated by a sufficient margin, $\sigma_{comp}(a,b)$ will be either [math] or $1$. The parameter $\lambda$ is used to control the precision of the comparator.

This function may be used to approximate the relative rank of two components $\mathbf{y}_{i}$ and $\mathbf{y}_{j}$ in a vector $\mathbf{y}$: $\sigma_{comp}(\mathbf{y}_{i},\mathbf{y}_{j})$ will be close to $1$ if $\mathbf{y}_{i}$ is (significantly) smaller than $\mathbf{y}_{j}$, 0 otherwise. By summing up the result of the comparison between $\mathbf{y}_{i}$ and all the other elements of the vector $\mathbf{y}$, we form our ranking function $f_{h}$. More precisely, the rank $f_{h}(\mathbf{y},i)$ for the $i$-est element of $\mathbf{y}$ is expressed as follow:

[TABLE]

The overall precision of the handcrafted sorter can be controlled by the hyper parameter $\lambda$. The value of lambda is a trade off between the precision of the predicted rank and the efficiency when back-propagating through the sorter. Further experiments will use $\lambda=10$.

4.1.3 Results

Table 1 contains the loss values of the two different trained sorters and the handcrafted one on a generated test set of 10 000 samples. The LSTM based sorter is the most efficient, outperforming the CNN and the handcrafted sorters.

The performance of the CNN sorter slightly below the LSTM-based one can be explained by local behaviour of the CNNs, requiring a more complex structure to be able to rank elements.

In Figure 4 we compare CNN sorters with respect to their number of layers. From these results, we choose to use 8 layers in our CNN sorter since the performance seems to saturate once this depth has been reached. A possible explanation of this saturation is that the relation between the depth of the network and the input dimension ($d=100$ here) is logarithmic.

4.1.4 Further analysis

The ranking function being non-continuous is non-differentiable, the rank value is jumping from one discrete value to another. We design an experiment to visualize how the different types of sorter behave at these discontinuities. Starting from a uniformly sampled vector $\mathbf{y}^{\prime}\in\mathbb{R}^{100}$ of raw scores in the range $[-1,1]$, we compute the ground truth rank $\mathbf{rk}(\mathbf{y}^{\prime})_{1}$ and the predicted rank $f_{\boldsymbol{\Theta}_{B}}(\mathbf{y}^{\prime})_{1}$ of the first element $y^{\prime}_{1}$ while varying this element $y^{\prime}_{1}$ from -1 to 1 in increments of 0.001. The plot of the predicted ranks can be found in Fig. 5. The blue curve corresponds to the ground-truth rank where non-continuous steps are visible, whereas the curves for the learned sorters (orange and green) are a smooth approximation of the ground-truth curve.

In Fig. 6 we compare our SoDeep against previous approaches optimizing structured hinge upper bound to the mAP loss. We followed the protocol described in [36] for their synthetic data experiments. Our sorters using the loss $\mathcal{L}_{mAP}$ defined in (7) are compared to a re-implementation of the Hinge-AP loss proposed in [21]. The results in Fig. 6 show that our approach with the LSTM sorter (blue curve) gets mAP scores similar to [21] (purple curve) while being generic and less complex.

From the learned sorters, the LSTM architecture is the one performing best on synthetic data (Tab. 1). In addition, its simple design and small number of hyper-parameters make it straightforward to train. The CNN architecture while not being as efficient, uses a smaller number of weights and is 1.7 time faster. Further experiments will use the LSTM sorter unless specified otherwise.

4.2 Differentiable Sorter based loss functions

Our method is benchmarked on three tasks. Each one of these tasks focuses on a different rank based loss function. Cross-modal retrieval will be used to test recall evaluation metrics, memorability prediction will be used for Spearman correlation and image classification will be used for mean average precision.

As explained in Section 3.1.2, a shift in distribution might appear when using sorter-based loss. To prevent this, a parallel loss can be used to help domain alignment. This loss can be used only to stabilize the initialization or kept for the whole training.

4.2.1 Spearman Correlation: Predicting Media Memorability

The media memorability prediction task [5] is used to test the differentiable sorter with respect to the Spearman correlation metrics. Examples of elements of the dataset can be found in Fig. 7. Given a 7 seconds video the task consists in predicting the short term memorability score. The memorability score reflects the probability of a video being remembered.

The task is originally on video memorability. However the model used here are pretrained on images, therefore 7 frames are extracted from each video and are associated with the memorability score of the source video. The training is done on pairs of frame and memorability score. During testing the predicted score of the 7 frames of a video are averaged to obtain the score per video. The dataset contains 8000 videos (56000 frames) for training and 2000 videos for testing. This training set is completed using LaMem dataset [23] adding 60 000 (image, memorability) pairs to the training data.

Architectures and training

The regression model consists of a feature extractor combined with a two layers MLP [33] regressing features to a single memorability score. We use two pretrained nets to extract features: the Resnet-34 [18] and the semantic embedding model of [10] (as in the next section).

We use the loss $\mathcal{L}_{SPR}$ defined in (4) to learn the memorability model. The training is done in two steps. First, for 15 epochs only the MLP layers are trained while the weights of the feature extractor are kept frozen. Second, the whole model is finetuned. The Adam optimizer [24] is used with a learning rate of $0.001$ which is halved every 3 epochs. To help with domain adaptation, our loss is combined with an L1 loss for the first epoch.

Results

In Tab. 2, we compare the impact of the learned loss over two architectures. For both models we defined a baseline using a L2 loss. On both architectures the proposed loss function achieves higher Spearman correlation by 2.4 points on the Resnet model and 0.8 points on the semantic embedding model. These are state of the arts result on the task with an absolute gain of 0.6 pt. The model is almost on par (-0.3 pt) with an ensemble method proposed by [17] that is using additional textual data.

Sorter comparison

The memorability prediction is also used to compare the different types of sorters presented so far. Fixing the model and the hyper parameters, 4 models are trained with 4 different types of loss. The losses based on the LSTM sorter, the CNN sorter and the handcrafted sorter obtained respectively a Spearman correlation of 49.4, 46.6, 45.7, and the L1 loss gives a correlation of 46.2. These results are consistent with the result on synthetic data, with the LSTM sorter performing the best, followed by the CNN and handcrafted ones.

4.2.2 Mean Average precision: Image classification

The VOC 2007 [11] object recognition challenge is used to evaluate our sorter on a task using the mean average precision metric. We use an off-the-shelf model [9]. This model is a fully convolutional network, combining a Resnet-101 [18] with advanced spatial aggregation mechanisms.

To evaluate the loss $\mathcal{L}_{mAP}$ defined in (7) two versions of the model are trained: A baseline using only multi-label soft margin loss, and another model trained using the multi-label soft margin loss combined with $\mathcal{L}_{mAP}$.

Rows 3 and 4 of Tab. 4 show the results obtained by the two previously described models. Both models are below the state-of-the-art, however the use of the rank loss is beneficial and improves the mAP by 0.8 pt compared to the model using only the soft margin loss.

4.2.3 Recall@K: Cross-modal Retrieval

The last benchmark used to evaluate the differentiable sorter is the cross-modal retrieval. Starting from images annotated with text, we train a model producing rich features for both image and text that live in the same embedding space. Similarity in the embedding space is then used to evaluate the quality of the model on the cross-modal retrieval task.

Our approach is evaluated on the MS-COCO dataset  [28] using the rVal split proposed in [22]. The dataset contains 110k images for training, 5k for validation and 5k for testing. Each image is annotated with 5 captions.

Given a query image (resp. a caption), the aim is to retrieve the corresponding captions (resp. image). Since MS-COCO contains 5 captions per image, recall at $r$ (“R@$r$”) for caption retrieval is computed based on whether at least one of the correct captions is among the first $r$ retrieved ones. The task is performed 5 times on 1000-image subsets of the test set and the results are averaged.

We use an off-the-shelf model [10]. It is a two-paths multimodal embedding approach that leverages the latest neural network architecture. The visual pipeline is based on a Resnet-152 and is fully convolutional. The textual pipeline is trained from scratch and uses a Simple Recurrent Unit (SRU) [27] to encode sentences. The model is trained using the loss $\mathcal{L}_{REC}$ defined in (10) instead of the triplet based loss.

Cross-modal retrieval results can be found in Tab. 3. The model trained using the proposed loss function (DSVE-Loc + SoDeep loss) outperforms the similar architecture DSVE-Loc trained with the triplet margin based loss by (1.7%,0.9%,0.5%) on (R@1,R@5,R@10) in absolute for caption retrieval, and by (0.3%,0.1%,0.3%) for image retrieval. It obtains state-of-the-art performance on caption retrieval and is very competitive on image retrieval being almost on par with the GXN [16] model, which has a much more complex architecture. It is important to note that the loss function proposed could be beneficial for any type of architecture.

5 Conclusion

We have presented SoDeep, a novel method that leverages the expressivity of recent architectures to learn differentiable surrogate functions. Based on a direct deep network modeling of the sorting operation, such a surrogate allows us to train, in an end-to-end manner, models on a diversity of tasks that are traditionally evaluated with rank-based metrics. Remarkably, this deep proxy to estimate the rank comes at virtually no cost since it is easily trained on purely synthetic data.

Our experiments show that the proposed approach achieves very good performance on cross-modal retrieval tasks as well as on media memorability prediction and multi-label image classification. These experiments demonstrate the potential and the versatility of SoDeep. This approach allows the design of training losses that are closer than before to metrics of interest, which opens up a wide range of other applications in the future.