Dissecting the impact of different loss functions with gradient surgery

TL;DR

This paper analyzes various pair-wise loss functions in metric learning by decomposing their gradients, unifies many existing methods, and introduces a simple algorithm that improves image retrieval performance.

Contribution

It provides a gradient decomposition framework that unifies and clarifies the effects of different pair-wise loss functions, leading to a new effective algorithm.

Findings

The new algorithm outperforms state-of-the-art on CAR, CUB, and Stanford Online datasets.

Gradient effects significantly influence the effectiveness of pair-wise loss functions.

Decomposition offers insights for designing better loss functions in metric learning.

Abstract

Pair-wise loss is an approach to metric learning that learns a semantic embedding by optimizing a loss function that encourages images from the same semantic class to be mapped closer than images from different classes. The literature reports a large and growing set of variations of the pair-wise loss strategies. Here we decompose the gradient of these loss functions into components that relate to how they push the relative feature positions of the anchor-positive and anchor-negative pairs. This decomposition allows the unification of a large collection of current pair-wise loss functions. Additionally, explicitly constructing pair-wise gradient updates to separate out these effects gives insights into which have the biggest impact, and leads to a simple algorithm that beats the state of the art for image retrieval on the CAR, CUB and Stanford Online products datasets.