Multi-level Supervised Contrastive Learning

TL;DR

This paper introduces a multilevel supervised contrastive learning framework that employs multiple projection heads to better capture complex similarities in data, improving performance on multi-label and hierarchical tasks.

Contribution

The paper proposes a novel multilevel contrastive learning method that utilizes multiple projection heads to capture diverse similarity aspects, enhancing representation learning.

Findings

Outperforms state-of-the-art contrastive methods on text and image datasets.

Effective in multi-label and hierarchical classification scenarios.

Improves representation quality with multiple projection heads.

Abstract

Contrastive learning is a well-established paradigm in representation learning. The standard framework of contrastive learning minimizes the distance between "similar" instances and maximizes the distance between dissimilar ones in the projection space, disregarding the various aspects of similarity that can exist between two samples. Current methods rely on a single projection head, which fails to capture the full complexity of different aspects of a sample, leading to suboptimal performance, especially in scenarios with limited training data. In this paper, we present a novel supervised contrastive learning method in a unified framework called multilevel contrastive learning (MLCL), that can be applied to both multi-label and hierarchical classification tasks. The key strength of the proposed method is the ability to capture similarities between samples across different labels and/or hierarchies using multiple projection heads. Extensive experiments on text and image datasets demonstrate that the proposed approach outperforms state-of-the-art contrastive learning methods