BiomedCoOp: Learning to Prompt for Biomedical Vision-Language Models

TL;DR

BiomedCoOp introduces a prompt learning framework that enhances biomedical vision-language models' accuracy and generalizability in few-shot image classification by leveraging semantic consistency and knowledge distillation.

Contribution

The paper presents a novel prompt learning method tailored for biomedical images, improving adaptation and performance of vision-language models in limited data scenarios.

Findings

Significant accuracy improvements over state-of-the-art methods.

Enhanced generalizability across multiple biomedical datasets.

Effective prompt context learning with semantic consistency and knowledge distillation.

Abstract

Recent advancements in vision-language models (VLMs), such as CLIP, have demonstrated substantial success in self-supervised representation learning for vision tasks. However, effectively adapting VLMs to downstream applications remains challenging, as their accuracy often depends on time-intensive and expertise-demanding prompt engineering, while full model fine-tuning is costly. This is particularly true for biomedical images, which, unlike natural images, typically suffer from limited annotated datasets, unintuitive image contrasts, and nuanced visual features. Recent prompt learning techniques, such as Context Optimization (CoOp) intend to tackle these issues, but still fall short in generalizability. Meanwhile, explorations in prompt learning for biomedical image analysis are still highly limited. In this work, we propose BiomedCoOp, a novel prompt learning framework that enables efficient adaptation of BiomedCLIP for accurate and highly generalizable few-shot biomedical image classification. Our approach achieves effective prompt context learning by leveraging semantic consistency with average prompt ensembles from Large Language Models (LLMs) and knowledge distillation with a statistics-based prompt selection strategy. We conducted comprehensive validation of our proposed framework on 11 medical datasets across 9 modalities and 10 organs against existing state-of-the-art methods, demonstrating significant improvements in both accuracy and generalizability. The code is publicly available at https://github.com/HealthX-Lab/BiomedCoOp.