Learning the Relation between Similarity Loss and Clustering Loss in Self-Supervised Learning

TL;DR

This paper explores how combining similarity loss and feature-level cross-entropy loss can enhance self-supervised learning, providing theoretical insights and empirical results that achieve state-of-the-art performance.

Contribution

It analyzes the relationship between similarity loss and cross-entropy loss and demonstrates that their optimal combination improves self-supervised learning performance.

Findings

Theoretical analysis clarifies the relation between the two losses.

Combining the two losses yields state-of-the-art results.

Experimental validation confirms the effectiveness of the proposed approach.

Abstract

Self-supervised learning enables networks to learn discriminative features from massive data itself. Most state-of-the-art methods maximize the similarity between two augmentations of one image based on contrastive learning. By utilizing the consistency of two augmentations, the burden of manual annotations can be freed. Contrastive learning exploits instance-level information to learn robust features. However, the learned information is probably confined to different views of the same instance. In this paper, we attempt to leverage the similarity between two distinct images to boost representation in self-supervised learning. In contrast to instance-level information, the similarity between two distinct images may provide more useful information. Besides, we analyze the relation between similarity loss and feature-level cross-entropy loss. These two losses are essential for most deep learning methods. However, the relation between these two losses is not clear. Similarity loss helps obtain instance-level representation, while feature-level cross-entropy loss helps mine the similarity between two distinct images. We provide theoretical analyses and experiments to show that a suitable combination of these two losses can get state-of-the-art results. Code is available at https://github.com/guijiejie/ICCL.