Freeze the backbones: A Parameter-Efficient Contrastive Approach to Robust Medical Vision-Language Pre-training

TL;DR

This paper proposes a parameter-efficient contrastive learning framework for medical vision-language pre-training that preserves pre-trained encoders and significantly reduces training costs while maintaining high performance.

Contribution

It introduces a backbone-agnostic Adaptor framework that keeps pre-trained encoders frozen and employs a lightweight module for cross-modal learning, reducing training parameters by over 90%.

Findings

Competitive performance on medical image classification and segmentation.

Outperforms full-data Transformer methods with only 1% of training data.

Reduces training computation significantly while maintaining accuracy.

Abstract

Modern healthcare often utilises radiographic images alongside textual reports for diagnostics, encouraging the use of Vision-Language Self-Supervised Learning (VL-SSL) with large pre-trained models to learn versatile medical vision representations. However, most existing VL-SSL frameworks are trained end-to-end, which is computation-heavy and can lose vital prior information embedded in pre-trained encoders. To address both issues, we introduce the backbone-agnostic Adaptor framework, which preserves medical knowledge in pre-trained image and text encoders by keeping them frozen, and employs a lightweight Adaptor module for cross-modal learning. Experiments on medical image classification and segmentation tasks across three datasets reveal that our framework delivers competitive performance while cutting trainable parameters by over 90% compared to current pre-training approaches. Notably, when fine-tuned with just 1% of data, Adaptor outperforms several Transformer-based methods trained on full datasets in medical image segmentation.