MedFocusCLIP : Improving few shot classification in medical datasets using pixel wise attention

TL;DR

This paper introduces MedFocusCLIP, a method that enhances few-shot medical image classification by guiding CLIP's attention to relevant regions using segmentation cues from SAM2, improving accuracy and interpretability.

Contribution

It proposes leveraging SAM2 segmentation as a visual prompt to improve CLIP's fine-grained medical image classification in few-shot settings.

Findings

Achieved higher accuracy on multiple medical datasets compared to baseline CLIP.

Enabled interpretable localization of discriminative regions in images.

Demonstrated effectiveness across X-ray, CT, and MRI datasets.

Abstract

With the popularity of foundational models, parameter efficient fine tuning has become the defacto approach to leverage pretrained models to perform downstream tasks. Taking inspiration from recent advances in large language models, Visual Prompt Tuning, and similar techniques, learn an additional prompt to efficiently finetune a pretrained vision foundational model. However, we observe that such prompting is insufficient for fine-grained visual classification tasks such as medical image classification, where there is large inter-class variance, and small intra-class variance. Hence, in this paper we propose to leverage advanced segmentation capabilities of Segment Anything Model 2 (SAM2) as a visual prompting cue to help visual encoder in the CLIP (Contrastive Language-Image Pretraining) by guiding the attention in CLIP visual encoder to relevant regions in the image. This helps the model to focus on highly discriminative regions, without getting distracted from visually similar background features, an essential requirement in a fewshot, finegrained classification setting. We evaluate our method on diverse medical datasets including X-rays, CT scans, and MRI images, and report an accuracy of (71%, 81%, 86%, 58%) from the proposed approach on (COVID, lung-disease, brain-tumor, breast-cancer) datasets against (66%, 70%, 68%, 29%) from a pretrained CLIP model after fewshot training. The proposed approach also allows to obtain interpretable explanation for the classification performance through the localization obtained using segmentation.