Seed the Views: Hierarchical Semantic Alignment for Contrastive Representation Learning

TL;DR

This paper introduces CsMl, a hierarchical semantic alignment method for contrastive learning that expands positive samples and aligns semantically similar images at multiple levels, improving representation quality and achieving state-of-the-art results.

Contribution

It proposes a novel hierarchical semantic alignment strategy that extends contrastive loss to incorporate multiple positives and multi-level semantic relationships, enhancing existing contrastive learning methods.

Findings

Achieves 76.6% top-1 accuracy with linear evaluation on ResNet-50.

Improves semi-supervised learning accuracy with only 1% and 10% labels.

Sets new state-of-the-art performance in contrastive representation learning.

Abstract

Self-supervised learning based on instance discrimination has shown remarkable progress. In particular, contrastive learning, which regards each image as well as its augmentations as an individual class and tries to distinguish them from all other images, has been verified effective for representation learning. However, pushing away two images that are de facto similar is suboptimal for general representation. In this paper, we propose a hierarchical semantic alignment strategy via expanding the views generated by a single image to \textbf{Cross-samples and Multi-level} representation, and models the invariance to semantically similar images in a hierarchical way. This is achieved by extending the contrastive loss to allow for multiple positives per anchor, and explicitly pulling semantically similar images/patches together at different layers of the network. Our method, termed as CsMl, has the ability to integrate multi-level visual representations across samples in a robust way. CsMl is applicable to current contrastive learning based methods and consistently improves the performance. Notably, using the moco as an instantiation, CsMl achieves a \textbf{76.6\% }top-1 accuracy with linear evaluation using ResNet-50 as backbone, and \textbf{66.7\%} and \textbf{75.1\%} top-1 accuracy with only 1\% and 10\% labels, respectively. \textbf{All these numbers set the new state-of-the-art.}