Variance & Greediness: A comparative study of metric-learning losses

TL;DR

This paper introduces a diagnostic framework to compare various metric-learning losses, revealing their effects on embedding geometry, convergence, and retrieval performance across datasets.

Contribution

The study provides a comprehensive comparison of seven metric-learning losses using the VARIANCE and GREEDINESS diagnostics, offering practical guidance for selecting losses based on task needs.

Findings

Triplet and SCL preserve higher within-class variance and inter-class margins.

Contrastive and InfoNCE achieve quick embedding compactness but may oversimplify class structures.

N-pair achieves large separation with uneven class spacing.

Abstract

Metric learning is central to retrieval, yet its effects on embedding geometry and optimization dynamics are not well understood. We introduce a diagnostic framework, VARIANCE (intra-/inter-class variance) and GREEDINESS (active ratio and gradient norms), to compare seven representative losses, i.e., Contrastive, Triplet, N-pair, InfoNCE, ArcFace, SCL, and CCL, across five image-retrieval datasets. Our analysis reveals that Triplet and SCL preserve higher within-class variance and clearer inter-class margins, leading to stronger top-1 retrieval in fine-grained settings. In contrast, Contrastive and InfoNCE compact embeddings are achieved quickly through many small updates, accelerating convergence but potentially oversimplifying class structures. N-pair achieves a large mean separation but with uneven spacing. These insights reveal a form of efficiency-granularity trade-off and provide practical guidance: prefer Triplet/SCL when diversity preservation and hard-sample discrimination are critical, and Contrastive/InfoNCE when faster embedding compaction is desired.