MedP-CLIP: Medical CLIP with Region-Aware Prompt Integration

TL;DR

MedP-CLIP is a region-aware medical vision-language model that enhances fine-grained understanding of anatomical regions in medical images through region prompt integration and large-scale pre-training.

Contribution

It introduces a novel feature-level region prompt mechanism and integrates medical prior knowledge, enabling flexible regional focus while maintaining global context in medical image analysis.

Findings

Outperforms baseline methods in zero-shot recognition, segmentation, and multimodal tasks.

Pre-trained on over 6.4 million images with 97.3 million annotations.

Demonstrates significant improvements in medical image understanding and regional analysis.

Abstract

Contrastive Language-Image Pre-training (CLIP) has demonstrated outstanding performance in global image understanding and zero-shot transfer through large-scale text-image alignment. However, the core of medical image analysis often lies in the fine-grained understanding of specific anatomical structures or lesion regions. Therefore, precisely comprehending region-of-interest (RoI) information provided by medical professionals or perception models becomes crucial. To address this need, we propose MedP-CLIP, a region-aware medical vision-language model (VLM). MedP-CLIP innovatively integrates medical prior knowledge and designs a feature-level region prompt integration mechanism, enabling it to flexibly respond to various prompt forms (e.g., points, bounding boxes, masks) while maintaining global contextual awareness when focusing on local regions. We pre-train the model on a meticulously constructed large-scale dataset (containing over 6.4 million medical images and 97.3 million region-level annotations), equipping it with cross-disease and cross-modality fine-grained spatial semantic understanding capabilities. Experiments demonstrate that MedP-CLIP significantly outperforms baseline methods in various medical tasks, including zero-shot recognition, interactive segmentation, and empowering multimodal large language models. This model provides a scalable, plug-and-play visual backbone for medical AI, combining holistic image understanding with precise regional analysis.