Boosting Memory Efficiency in Transfer Learning for High-Resolution Medical Image Classification

TL;DR

This paper introduces FPT+, a parameter-efficient transfer learning method that significantly reduces memory consumption for high-resolution medical image classification by using a lightweight side network with fine-grained prompts, outperforming existing PETL methods.

Contribution

FPT+ is a novel PETL approach that leverages a frozen large pre-trained model with a lightweight side network and fine-grained prompts to enable high-resolution medical image classification with minimal memory usage.

Findings

FPT+ uses only 1.03% of learnable parameters compared to full fine-tuning.

FPT+ requires only 3.18% of the memory needed for full model fine-tuning.

FPT+ outperforms other PETL methods across eight diverse medical image datasets.

Abstract

The success of large-scale pre-trained models has established fine-tuning as a standard method for achieving significant improvements in downstream tasks. However, fine-tuning the entire parameter set of a pre-trained model is costly. Parameter-efficient transfer learning (PETL) has recently emerged as a cost-effective alternative for adapting pre-trained models to downstream tasks. Despite its advantages, the increasing model size and input resolution present challenges for PETL, as the training memory consumption is not reduced as effectively as the parameter usage. In this paper, we introduce Fine-grained Prompt Tuning plus (FPT+), a PETL method designed for high-resolution medical image classification, which significantly reduces the training memory consumption compared to other PETL methods. FPT+ performs transfer learning by training a lightweight side network and accessing pre-trained knowledge from a large pre-trained model (LPM) through fine-grained prompts and fusion modules. Specifically, we freeze the LPM of interest and construct a learnable lightweight side network. The frozen LPM processes high-resolution images to extract fine-grained features, while the side network employs corresponding down-sampled low-resolution images to minimize the memory usage. To enable the side network to leverage pre-trained knowledge, we propose fine-grained prompts and fusion modules, which collaborate to summarize information through the LPM's intermediate activations. We evaluate FPT+ on eight medical image datasets of varying sizes, modalities, and complexities. Experimental results demonstrate that FPT+ outperforms other PETL methods, using only 1.03% of the learnable parameters and 3.18% of the memory required for fine-tuning an entire ViT-B model. Our code is available on https://github.com/YijinHuang/FPT.