Rethinking Deep Contrastive Learning with Embedding Memory

TL;DR

This paper introduces a systematic approach to analyze pair-wise loss functions in deep metric learning using embedding memory, emphasizing the importance of mining hard negatives and simplifying weighting schemes for improved performance.

Contribution

It proposes a new methodology for studying pair weighting strategies in memory-based deep metric learning, revealing insights that lead to a simple yet effective weighting rule.

Findings

Memory-based DML effectively mines hard negatives.

Weighting positive pairs is less beneficial than focusing on negatives.

Simple contrastive loss with memory outperforms complex mini-batch methods.

Abstract

Pair-wise loss functions have been extensively studied and shown to continuously improve the performance of deep metric learning (DML). However, they are primarily designed with intuition based on simple toy examples, and experimentally identifying the truly effective design is difficult in complicated, real-world cases. In this paper, we provide a new methodology for systematically studying weighting strategies of various pair-wise loss functions, and rethink pair weighting with an embedding memory. We delve into the weighting mechanisms by decomposing the pair-wise functions, and study positive and negative weights separately using direct weight assignment. This allows us to study various weighting functions deeply and systematically via weight curves, and identify a number of meaningful, comprehensive and insightful facts, which come up with our key observation on memory-based DML: it is critical to mine hard negatives and discard easy negatives which are less informative and redundant, but weighting on positive pairs is not helpful. This results in an efficient but surprisingly simple rule to design the weighting scheme, making it significantly different from existing mini-batch based methods which design various sophisticated loss functions to weight pairs carefully. Finally, we conduct extensive experiments on three large-scale visual retrieval benchmarks, and demonstrate the superiority of memory-based DML over recent mini-batch based approaches, by using a simple contrastive loss with momentum-updated memory.