Rethinking Transfer Learning for Medical Image Classification

TL;DR

This paper introduces TruncatedTL, a new transfer learning strategy for medical image classification that selectively reuses and discards layers, leading to better performance and more compact models.

Contribution

It proposes TruncatedTL, a novel differential transfer learning method that improves MIC accuracy and model efficiency by selectively truncating pretrained layers.

Findings

TruncatedTL outperforms existing differential TL methods in MIC tasks.

It produces more compact models with comparable or better accuracy.

The approach enhances inference efficiency in medical imaging applications.

Abstract

Transfer learning (TL) from pretrained deep models is a standard practice in modern medical image classification (MIC). However, what levels of features to be reused are problem-dependent, and uniformly finetuning all layers of pretrained models may be suboptimal. This insight has partly motivated the recent differential TL strategies, such as TransFusion (TF) and layer-wise finetuning (LWFT), which treat the layers in the pretrained models differentially. In this paper, we add one more strategy into this family, called TruncatedTL, which reuses and finetunes appropriate bottom layers and directly discards the remaining layers. This yields not only superior MIC performance but also compact models for efficient inference, compared to other differential TL methods. Our code is available at: https://github.com/sun-umn/TTL