Optimization of Rank Losses for Image Retrieval

TL;DR

This paper introduces a novel framework for optimizing rank-based loss functions in image retrieval, addressing non-differentiability and non-decomposability issues to improve training robustness and performance.

Contribution

It proposes SupRank, a surrogate for ranking operators, and a loss to bridge the decomposability gap, enabling end-to-end training for ranking metrics like AP and R@k.

Findings

SupRank provides a robust upper bound for rank losses.

The framework improves optimization of AP and R@k metrics.

A new hierarchical landmarks dataset was created for evaluation.

Abstract

In image retrieval, standard evaluation metrics rely on score ranking, \eg average precision (AP), recall at k (R@k), normalized discounted cumulative gain (NDCG). In this work we introduce a general framework for robust and decomposable rank losses optimization. It addresses two major challenges for end-to-end training of deep neural networks with rank losses: non-differentiability and non-decomposability. Firstly we propose a general surrogate for ranking operator, SupRank, that is amenable to stochastic gradient descent. It provides an upperbound for rank losses and ensures robust training. Secondly, we use a simple yet effective loss function to reduce the decomposability gap between the averaged batch approximation of ranking losses and their values on the whole training set. We apply our framework to two standard metrics for image retrieval: AP and R@k. Additionally we apply our framework to hierarchical image retrieval. We introduce an extension of AP, the hierarchical average precision $\mathcal{H}$-AP, and optimize it as well as the NDCG. Finally we create the first hierarchical landmarks retrieval dataset. We use a semi-automatic pipeline to create hierarchical labels, extending the large scale Google Landmarks v2 dataset. The hierarchical dataset is publicly available at https://github.com/cvdfoundation/google-landmark. Code will be released at https://github.com/elias-ramzi/SupRank.