Intriguing Properties of Contrastive Losses

TL;DR

This paper explores three intriguing properties of contrastive learning, including a generalized loss framework, the ability to learn hierarchical features in multi-object images, and the phenomenon of feature suppression among shared features, highlighting challenges and open questions.

Contribution

The paper introduces a generalized contrastive loss, demonstrates hierarchical feature learning in multi-object images, and analyzes feature suppression effects, revealing limitations of current contrastive methods.

Findings

Generalized contrastive loss performs similarly across different instantiations.

Contrastive learning can learn hierarchical features despite global objectives.

Shared features can suppress learning of other features, affecting performance.

Abstract

We study three intriguing properties of contrastive learning. First, we generalize the standard contrastive loss to a broader family of losses, and we find that various instantiations of the generalized loss perform similarly under the presence of a multi-layer non-linear projection head. Second, we study if instance-based contrastive learning (with a global image representation) can learn well on images with multiple objects present. We find that meaningful hierarchical local features can be learned despite the fact that these objectives operate on global instance-level features. Finally, we study the phenomenon of feature suppression among competing features shared across augmented views, such as "color distribution" vs "object class". We construct datasets with explicit and controllable competing features, and show that, for contrastive learning, a few bits of easy-to-learn shared features can suppress, and even fully prevent, the learning of other sets of competing features. In scenarios where there are multiple objects in an image, the dominant object would suppress the learning of smaller objects. Existing contrastive learning methods critically rely on data augmentation to favor certain sets of features over others, and could suffer from learning saturation for scenarios where existing augmentations cannot fully address the feature suppression. This poses open challenges to existing contrastive learning techniques.