Smooth-AP: Smoothing the Path Towards Large-Scale Image Retrieval

TL;DR

This paper introduces Smooth-AP, a differentiable approximation of Average Precision that enables end-to-end training of deep networks for large-scale image retrieval, outperforming existing methods on multiple benchmarks.

Contribution

The paper proposes Smooth-AP, a novel differentiable objective for optimizing AP directly, facilitating scalable and effective deep metric learning for image retrieval.

Findings

Smooth-AP improves retrieval performance on multiple benchmarks.

It is especially effective on large-scale datasets.

Smooth-AP enables end-to-end training with simple implementation.

Abstract

Optimising a ranking-based metric, such as Average Precision (AP), is notoriously challenging due to the fact that it is non-differentiable, and hence cannot be optimised directly using gradient-descent methods. To this end, we introduce an objective that optimises instead a smoothed approximation of AP, coined Smooth-AP. Smooth-AP is a plug-and-play objective function that allows for end-to-end training of deep networks with a simple and elegant implementation. We also present an analysis for why directly optimising the ranking based metric of AP offers benefits over other deep metric learning losses. We apply Smooth-AP to standard retrieval benchmarks: Stanford Online products and VehicleID, and also evaluate on larger-scale datasets: INaturalist for fine-grained category retrieval, and VGGFace2 and IJB-C for face retrieval. In all cases, we improve the performance over the state-of-the-art, especially for larger-scale datasets, thus demonstrating the effectiveness and scalability of Smooth-AP to real-world scenarios.