Multi-label Cluster Discrimination for Visual Representation Learning

TL;DR

This paper introduces MLCD, a multi-label cluster discrimination method that improves visual representation learning by capturing multiple semantic signals in images, outperforming previous single-label approaches.

Contribution

The paper proposes a novel multi-label cluster discrimination technique that leverages multiple pseudo-labels per image to better encode semantic structures in visual representations.

Findings

Achieves state-of-the-art results on downstream tasks

Effectively captures multi-object and attribute signals in images

Improves over single-label clustering methods

Abstract

Contrastive Language Image Pre-training (CLIP) has recently demonstrated success across various tasks due to superior feature representation empowered by image-text contrastive learning. However, the instance discrimination method used by CLIP can hardly encode the semantic structure of training data. To handle this limitation, cluster discrimination has been proposed through iterative cluster assignment and classification. Nevertheless, most cluster discrimination approaches only define a single pseudo-label for each image, neglecting multi-label signals in the image. In this paper, we propose a novel Multi-Label Cluster Discrimination method named MLCD to enhance representation learning. In the clustering step, we first cluster the large-scale LAION-400M dataset into one million centers based on off-the-shelf embedding features. Considering that natural images frequently contain multiple visual objects or attributes, we select the multiple closest centers as auxiliary class labels. In the discrimination step, we design a novel multi-label classification loss, which elegantly separates losses from positive classes and negative classes, and alleviates ambiguity on decision boundary. We validate the proposed multi-label cluster discrimination method with experiments on different scales of models and pre-training datasets. Experimental results show that our method achieves state-of-the-art performance on multiple downstream tasks including linear probe, zero-shot classification, and image-text retrieval. Code and models have been released at https://github.com/deepglint/unicom .