EFCM: Efficient Fine-tuning on Compressed Models for deployment of large models in medical image analysis

TL;DR

This paper introduces EFCM, a two-stage framework combining unsupervised feature distillation and fine-tuning to enable efficient deployment of large medical image analysis models, improving accuracy and inference speed.

Contribution

The study proposes a novel EFCM framework with Feature Projection Distillation and adaptive fine-tuning strategies for large medical models.

Findings

Achieved 4.33% accuracy increase on TCGA datasets.

Improved AUC by 5.2% over baseline models.

Demonstrated high inference efficiency of the proposed method.

Abstract

The recent development of deep learning large models in medicine shows remarkable performance in medical image analysis and diagnosis, but their large number of parameters causes memory and inference latency challenges. Knowledge distillation offers a solution, but the slide-level gradients cannot be backpropagated for student model updates due to high-resolution pathological images and slide-level labels. This study presents an Efficient Fine-tuning on Compressed Models (EFCM) framework with two stages: unsupervised feature distillation and fine-tuning. In the distillation stage, Feature Projection Distillation (FPD) is proposed with a TransScan module for adaptive receptive field adjustment to enhance the knowledge absorption capability of the student model. In the slide-level fine-tuning stage, three strategies (Reuse CLAM, Retrain CLAM, and End2end Train CLAM (ETC)) are compared. Experiments are conducted on 11 downstream datasets related to three large medical models: RETFound for retina, MRM for chest X-ray, and BROW for histopathology. The experimental results demonstrate that the EFCM framework significantly improves accuracy and efficiency in handling slide-level pathological image problems, effectively addressing the challenges of deploying large medical models. Specifically, it achieves a 4.33% increase in ACC and a 5.2% increase in AUC compared to the large model BROW on the TCGA-NSCLC and TCGA-BRCA datasets. The analysis of model inference efficiency highlights the high efficiency of the distillation fine-tuning method.