Solving Inefficiency of Self-supervised Representation Learning

TL;DR

This paper introduces a novel self-supervised learning framework using truncated triplet loss to address low learning efficiency caused by under-clustering and over-clustering in contrastive learning, demonstrating superior performance on large-scale benchmarks.

Contribution

The paper proposes a new self-supervised learning method with truncated triplet loss to improve efficiency by solving clustering issues in contrastive learning.

Findings

Outperforms state-of-the-art methods on ImageNet, SYSU-30k, and COCO.

Significantly reduces training epochs needed for comparable accuracy.

Effectively addresses under-clustering and over-clustering problems.

Abstract

Self-supervised learning (especially contrastive learning) has attracted great interest due to its huge potential in learning discriminative representations in an unsupervised manner. Despite the acknowledged successes, existing contrastive learning methods suffer from very low learning efficiency, e.g., taking about ten times more training epochs than supervised learning for comparable recognition accuracy. In this paper, we reveal two contradictory phenomena in contrastive learning that we call under-clustering and over-clustering problems, which are major obstacles to learning efficiency. Under-clustering means that the model cannot efficiently learn to discover the dissimilarity between inter-class samples when the negative sample pairs for contrastive learning are insufficient to differentiate all the actual object classes. Over-clustering implies that the model cannot efficiently learn features from excessive negative sample pairs, forcing the model to over-cluster samples of the same actual classes into different clusters. To simultaneously overcome these two problems, we propose a novel self-supervised learning framework using a truncated triplet loss. Precisely, we employ a triplet loss tending to maximize the relative distance between the positive pair and negative pairs to address the under-clustering problem; and we construct the negative pair by selecting a negative sample deputy from all negative samples to avoid the over-clustering problem, guaranteed by the Bernoulli Distribution model. We extensively evaluate our framework in several large-scale benchmarks (e.g., ImageNet, SYSU-30k, and COCO). The results demonstrate our model's superiority (e.g., the learning efficiency) over the latest state-of-the-art methods by a clear margin. Codes available at: https://github.com/wanggrun/triplet .