Improve Unsupervised Pretraining for Few-label Transfer

TL;DR

This paper investigates the limitations of unsupervised pretraining in few-label transfer scenarios, analyzes clustering quality issues, and proposes a progressive transfer method that leverages unlabeled target data to improve transfer performance.

Contribution

It reveals the importance of target sample clustering quality in few-label transfer and introduces a new progressive algorithm utilizing unlabeled target data to enhance transfer results.

Findings

Unsupervised pretraining's effectiveness diminishes with very few labels.

Involving unlabeled target data improves clustering and transfer performance.

Proposed method significantly outperforms existing approaches on multiple datasets.

Abstract

Unsupervised pretraining has achieved great success and many recent works have shown unsupervised pretraining can achieve comparable or even slightly better transfer performance than supervised pretraining on downstream target datasets. But in this paper, we find this conclusion may not hold when the target dataset has very few labeled samples for finetuning, \ie, few-label transfer. We analyze the possible reason from the clustering perspective: 1) The clustering quality of target samples is of great importance to few-label transfer; 2) Though contrastive learning is essential to learn how to cluster, its clustering quality is still inferior to supervised pretraining due to lack of label supervision. Based on the analysis, we interestingly discover that only involving some unlabeled target domain into the unsupervised pretraining can improve the clustering quality, subsequently reducing the transfer performance gap with supervised pretraining. This finding also motivates us to propose a new progressive few-label transfer algorithm for real applications, which aims to maximize the transfer performance under a limited annotation budget. To support our analysis and proposed method, we conduct extensive experiments on nine different target datasets. Experimental results show our proposed method can significantly boost the few-label transfer performance of unsupervised pretraining.