CLEFT: Language-Image Contrastive Learning with Efficient Large Language Model and Prompt Fine-Tuning

TL;DR

CLEFT introduces an efficient contrastive learning approach combining large language models and prompt fine-tuning, achieving state-of-the-art results in medical imaging with reduced computational resources.

Contribution

It presents a novel, resource-efficient contrastive learning framework that leverages large pre-trained language models and context-based prompt learning for medical image analysis.

Findings

State-of-the-art performance on chest X-ray and mammography datasets

Reduces trainable model size by 39% and language model parameters to 4%

Effective prompt learning strategy bridges clinical data and labels

Abstract

Recent advancements in Contrastive Language-Image Pre-training (CLIP) have demonstrated notable success in self-supervised representation learning across various tasks. However, the existing CLIP-like approaches often demand extensive GPU resources and prolonged training times due to the considerable size of the model and dataset, making them poor for medical applications, in which large datasets are not always common. Meanwhile, the language model prompts are mainly manually derived from labels tied to images, potentially overlooking the richness of information within training samples. We introduce a novel language-image Contrastive Learning method with an Efficient large language model and prompt Fine-Tuning (CLEFT) that harnesses the strengths of the extensive pre-trained language and visual models. Furthermore, we present an efficient strategy for learning context-based prompts that mitigates the gap between informative clinical diagnostic data and simple class labels. Our method demonstrates state-of-the-art performance on multiple chest X-ray and mammography datasets compared with various baselines. The proposed parameter efficient framework can reduce the total trainable model size by 39% and reduce the trainable language model to only 4% compared with the current BERT encoder.