Mathematical Justification of Hard Negative Mining via Isometric Approximation Theorem

TL;DR

This paper provides a mathematical justification for hard negative mining in deep metric learning by linking it to isometric approximation, explaining its empirical success and guiding future improvements.

Contribution

It introduces a theoretical framework using isometric approximation to justify hard negative mining and its effectiveness in preventing network collapse.

Findings

Hard negative mining is theoretically equivalent to minimizing a Hausdorff-like distance.

The approach explains the empirical success of hard negative mining.

Framework can extend to other Euclidean space-based metric learning methods.

Abstract

In deep metric learning, the Triplet Loss has emerged as a popular method to learn many computer vision and natural language processing tasks such as facial recognition, object detection, and visual-semantic embeddings. One issue that plagues the Triplet Loss is network collapse, an undesirable phenomenon where the network projects the embeddings of all data onto a single point. Researchers predominately solve this problem by using triplet mining strategies. While hard negative mining is the most effective of these strategies, existing formulations lack strong theoretical justification for their empirical success. In this paper, we utilize the mathematical theory of isometric approximation to show an equivalence between the Triplet Loss sampled by hard negative mining and an optimization problem that minimizes a Hausdorff-like distance between the neural network and its ideal counterpart function. This provides the theoretical justifications for hard negative mining's empirical efficacy. In addition, our novel application of the isometric approximation theorem provides the groundwork for future forms of hard negative mining that avoid network collapse. Our theory can also be extended to analyze other Euclidean space-based metric learning methods like Ladder Loss or Contrastive Learning.