Contrastive Learning Meets Transfer Learning: A Case Study In Medical Image Analysis

TL;DR

This paper explores combining transfer learning with contrastive learning in medical image analysis, demonstrating that transfer models like BiT can accelerate contrastive learning convergence and improve performance.

Contribution

It investigates the integration of Big Transfer (BiT) with SimSiam contrastive learning, highlighting normalization as a key challenge and showing improved convergence and accuracy.

Findings

BiT accelerates SimSiam convergence.

Combined BiT+SimSiam outperforms individual models.

Normalization techniques are crucial for integration.

Abstract

Annotated medical images are typically rarer than labeled natural images since they are limited by domain knowledge and privacy constraints. Recent advances in transfer and contrastive learning have provided effective solutions to tackle such issues from different perspectives. The state-of-the-art transfer learning (e.g., Big Transfer (BiT)) and contrastive learning (e.g., Simple Siamese Contrastive Learning (SimSiam)) approaches have been investigated independently, without considering the complementary nature of such techniques. It would be appealing to accelerate contrastive learning with transfer learning, given that slow convergence speed is a critical limitation of modern contrastive learning approaches. In this paper, we investigate the feasibility of aligning BiT with SimSiam. From empirical analyses, different normalization techniques (Group Norm in BiT vs. Batch Norm in SimSiam) are the key hurdle of adapting BiT to SimSiam. When combining BiT with SimSiam, we evaluated the performance of using BiT, SimSiam, and BiT+SimSiam on CIFAR-10 and HAM10000 datasets. The results suggest that the BiT models accelerate the convergence speed of SimSiam. When used together, the model gives superior performance over both of its counterparts. We hope this study will motivate researchers to revisit the task of aggregating big pre-trained models with contrastive learning models for image analysis.