Unleashing the Power of Contrastive Self-Supervised Visual Models via Contrast-Regularized Fine-Tuning

TL;DR

This paper introduces Contrast-regularized fine-tuning (Core-tuning), a novel method that enhances contrastive self-supervised visual models by applying contrastive loss with hard pair mining and boundary smoothing, improving downstream task performance.

Contribution

It proposes a new fine-tuning approach that leverages contrastive loss with hard pair mining and boundary smoothing to better utilize self-supervised features.

Findings

Core-tuning improves image classification accuracy.

Core-tuning enhances semantic segmentation performance.

Contrastive fine-tuning benefits downstream tasks.

Abstract

Contrastive self-supervised learning (CSL) has attracted increasing attention for model pre-training via unlabeled data. The resulted CSL models provide instance-discriminative visual features that are uniformly scattered in the feature space. During deployment, the common practice is to directly fine-tune CSL models with cross-entropy, which however may not be the best strategy in practice. Although cross-entropy tends to separate inter-class features, the resulting models still have limited capability for reducing intra-class feature scattering that exists in CSL models. In this paper, we investigate whether applying contrastive learning to fine-tuning would bring further benefits, and analytically find that optimizing the contrastive loss benefits both discriminative representation learning and model optimization during fine-tuning. Inspired by these findings, we propose Contrast-regularized tuning (Core-tuning), a new approach for fine-tuning CSL models. Instead of simply adding the contrastive loss to the objective of fine-tuning, Core-tuning further applies a novel hard pair mining strategy for more effective contrastive fine-tuning, as well as smoothing the decision boundary to better exploit the learned discriminative feature space. Extensive experiments on image classification and semantic segmentation verify the effectiveness of Core-tuning.