Fine-grained Prompt Tuning: A Parameter and Memory Efficient Transfer Learning Method for High-resolution Medical Image Classification

TL;DR

This paper introduces Fine-grained Prompt Tuning (FPT), a parameter-efficient transfer learning method for high-resolution medical image classification that significantly reduces memory and parameter usage while maintaining performance.

Contribution

FPT is a novel PETL approach that combines a frozen large pre-trained model with a lightweight side network and prompts, enabling efficient high-resolution medical image classification.

Findings

FPT achieves comparable accuracy to full fine-tuning.

Uses only 1.8% of parameters compared to full fine-tuning.

Reduces memory costs to 13% of a standard encoder model.

Abstract

Parameter-efficient transfer learning (PETL) is proposed as a cost-effective way to transfer pre-trained models to downstream tasks, avoiding the high cost of updating entire large-scale pre-trained models (LPMs). In this work, we present Fine-grained Prompt Tuning (FPT), a novel PETL method for medical image classification. FPT significantly reduces memory consumption compared to other PETL methods, especially in high-resolution input contexts. To achieve this, we first freeze the weights of the LPM and construct a learnable lightweight side network. The frozen LPM takes high-resolution images as input to extract fine-grained features, while the side network is fed low-resolution images to reduce memory usage. To allow the side network to access pre-trained knowledge, we introduce fine-grained prompts that summarize information from the LPM through a fusion module. Important tokens selection and preloading techniques are employed to further reduce training cost and memory requirements. We evaluate FPT on four medical datasets with varying sizes, modalities, and complexities. Experimental results demonstrate that FPT achieves comparable performance to fine-tuning the entire LPM while using only 1.8% of the learnable parameters and 13% of the memory costs of an encoder ViT-B model with a 512 x 512 input resolution.