Boosting Medical Vision-Language Pretraining via Momentum Self-Distillation under Limited Computing Resources

TL;DR

This paper introduces a momentum self-distillation method for medical vision-language pretraining that enhances performance and efficiency, enabling effective multimodal learning with limited computational resources and data.

Contribution

It proposes leveraging momentum self-distillation combined with gradient accumulation to improve multimodal learning efficiency and performance in healthcare applications.

Findings

Achieves over 90% AUC-ROC in zero-shot classification.

Boosts retrieval task performance by 2-3%.

Operates efficiently on a single GPU with reasonable training time.

Abstract

In medical healthcare, obtaining detailed annotations is challenging, highlighting the need for robust Vision-Language Models (VLMs). Pretrained VLMs enable fine-tuning on small datasets or zero-shot inference, achieving performance comparable to task-specific models. Contrastive learning (CL) is a key paradigm for training VLMs but inherently requires large batch sizes for effective learning, making it computationally demanding and often limited to well-resourced institutions. Moreover, with limited data in healthcare, it is important to prioritize knowledge extraction from both data and models during training to improve performance. Therefore, we focus on leveraging the momentum method combined with distillation to simultaneously address computational efficiency and knowledge exploitation. Our contributions can be summarized as follows: (1) leveraging momentum self-distillation to enhance multimodal learning, and (2) integrating momentum mechanisms with gradient accumulation to enlarge the effective batch size without increasing resource consumption. Our method attains competitive performance with state-of-the-art (SOTA) approaches in zero-shot classification, while providing a substantial boost in the few-shot adaption, achieving over 90% AUC-ROC and improving retrieval tasks by 2-3%. Importantly, our method achieves high training efficiency with a single GPU while maintaining reasonable training time. Our approach aims to advance efficient multimodal learning by reducing resource requirements while improving performance over SOTA methods. The implementation of our method is available at https://github.com/phphuc612/MSD .