HCSC: Hierarchical Contrastive Selective Coding

TL;DR

HCSC introduces a hierarchical contrastive learning framework that captures semantic structures in image data, improving representation quality for downstream tasks by better selecting positive and negative pairs.

Contribution

The paper proposes a novel hierarchical contrastive learning framework with dynamic prototypes and an elaborate pair selection scheme, addressing limitations of existing contrastive methods.

Findings

HCSC outperforms state-of-the-art contrastive methods on multiple downstream tasks.

The hierarchical prototypes effectively capture semantic structures in image data.

The pair selection scheme improves the diversity and semantic correctness of positive and negative pairs.

Abstract

Hierarchical semantic structures naturally exist in an image dataset, in which several semantically relevant image clusters can be further integrated into a larger cluster with coarser-grained semantics. Capturing such structures with image representations can greatly benefit the semantic understanding on various downstream tasks. Existing contrastive representation learning methods lack such an important model capability. In addition, the negative pairs used in these methods are not guaranteed to be semantically distinct, which could further hamper the structural correctness of learned image representations. To tackle these limitations, we propose a novel contrastive learning framework called Hierarchical Contrastive Selective Coding (HCSC). In this framework, a set of hierarchical prototypes are constructed and also dynamically updated to represent the hierarchical semantic structures underlying the data in the latent space. To make image representations better fit such semantic structures, we employ and further improve conventional instance-wise and prototypical contrastive learning via an elaborate pair selection scheme. This scheme seeks to select more diverse positive pairs with similar semantics and more precise negative pairs with truly distinct semantics. On extensive downstream tasks, we verify the superior performance of HCSC over state-of-the-art contrastive methods, and the effectiveness of major model components is proved by plentiful analytical studies. We build a comprehensive model zoo in Sec. D. Our source code and model weights are available at https://github.com/gyfastas/HCSC