Deep Reprogramming Distillation for Medical Foundation Models

TL;DR

This paper introduces Deep Reprogramming Distillation (DRD), a novel framework that enhances the adaptation of large medical foundation models to specific tasks by overcoming domain gaps and improving knowledge transfer efficiency.

Contribution

The paper proposes DRD with a reprogramming module and CKA distillation, enabling effective adaptation and robust knowledge transfer from foundation models to lightweight medical models.

Findings

DRD outperforms previous PEFT and KD methods on 18 medical tasks.

DRD effectively handles 2D/3D classification and segmentation scenarios.

Empirical results demonstrate superior performance across diverse medical applications.

Abstract

Medical foundation models pre-trained on large-scale datasets have shown powerful versatile performance. However, when adapting medical foundation models for specific medical scenarios, it remains the inevitable challenge due to the gap induced by the discrepancy between pre-training and downstream tasks, the real-world computation, and speed constraints. Relevant techniques that probably handle this challenge more or less suffer from some intrinsic limitations. For example, knowledge distillation (KD) assumes that teacher and student models share the same task, training strategy, and model structure family, while prevalent parameter-efficient fine-tuning (PEFT) fails to achieve personalized and lightweight deployment. Even the combination of PEFT and KD still struggles to resolve model structures and training strategies inconsistencies between teacher and student models, leading to inefficient knowledge transfer. In this study, we propose a novel framework called Deep Reprogramming Distillation (DRD) to combat the general adaptation challenge. Specifically, DRD introduces the novel reprogramming module that on the one side overcomes the domain and task discrepancy between pretraining and downstream scenarios, and on the other side builds the student-friendly efficient distillation from foundation models to lightweight downstream models. Furthermore, to mitigate variability under different training conditions, we design a centered kernel alignment (CKA) distillation method to promote robust knowledge transfer. Empirical results show that DRD surpasses previous PEFT and KD methods across 18 medical downstream tasks under different foundation models, covering various scenarios including 2D/3D classification and 2D/3D segmentation.