Train a One-Million-Way Instance Classifier for Unsupervised Visual Representation Learning

TL;DR

This paper introduces a scalable unsupervised visual representation learning method using a one-million-way instance classifier, addressing large-scale training challenges and achieving competitive results on ImageNet and downstream tasks.

Contribution

The work proposes novel techniques for large-scale instance classification, including hybrid parallel training, raw-feature weight initialization, and label smoothing for hard classes.

Findings

Achieves competitive performance on ImageNet linear evaluation.

Outperforms or matches state-of-the-art unsupervised methods.

Demonstrates effectiveness on multiple downstream visual tasks.

Abstract

This paper presents a simple unsupervised visual representation learning method with a pretext task of discriminating all images in a dataset using a parametric, instance-level classifier. The overall framework is a replica of a supervised classification model, where semantic classes (e.g., dog, bird, and ship) are replaced by instance IDs. However, scaling up the classification task from thousands of semantic labels to millions of instance labels brings specific challenges including 1) the large-scale softmax computation; 2) the slow convergence due to the infrequent visiting of instance samples; and 3) the massive number of negative classes that can be noisy. This work presents several novel techniques to handle these difficulties. First, we introduce a hybrid parallel training framework to make large-scale training feasible. Second, we present a raw-feature initialization mechanism for classification weights, which we assume offers a contrastive prior for instance discrimination and can clearly speed up converge in our experiments. Finally, we propose to smooth the labels of a few hardest classes to avoid optimizing over very similar negative pairs. While being conceptually simple, our framework achieves competitive or superior performance compared to state-of-the-art unsupervised approaches, i.e., SimCLR, MoCoV2, and PIC under ImageNet linear evaluation protocol and on several downstream visual tasks, verifying that full instance classification is a strong pretraining technique for many semantic visual tasks.