Simple and Effective Balance of Contrastive Losses

TL;DR

This paper presents a hyper-parameter optimization method for balancing contrastive losses in deep metric and self-supervised learning, improving performance and efficiency.

Contribution

It introduces a coordinate descent-based search for optimal contrastive loss hyper-parameters, including batch size, extending existing analyses and demonstrating faster convergence.

Findings

Faster hyper-parameter tuning compared to other methods

Effective balance of contrastive loss components improves performance

Applicable to various benchmarks in metric and self-supervised learning

Abstract

Contrastive losses have long been a key ingredient of deep metric learning and are now becoming more popular due to the success of self-supervised learning. Recent research has shown the benefit of decomposing such losses into two sub-losses which act in a complementary way when learning the representation network: a positive term and an entropy term. Although the overall loss is thus defined as a combination of two terms, the balance of these two terms is often hidden behind implementation details and is largely ignored and sub-optimal in practice. In this work, we approach the balance of contrastive losses as a hyper-parameter optimization problem, and propose a coordinate descent-based search method that efficiently find the hyper-parameters that optimize evaluation performance. In the process, we extend existing balance analyses to the contrastive margin loss, include batch size in the balance, and explain how to aggregate loss elements from the batch to maintain near-optimal performance over a larger range of batch sizes. Extensive experiments with benchmarks from deep metric learning and self-supervised learning show that optimal hyper-parameters are found faster with our method than with other common search methods.