QwenCLIP: Boosting Medical Vision-Language Pretraining via LLM Embeddings and Prompt tuning

TL;DR

QwenCLIP enhances medical vision-language pretraining by integrating large language models and prompt tuning, enabling better understanding of long radiology reports and improving alignment and performance on medical benchmarks.

Contribution

It introduces a novel framework replacing CLIP's text encoder with an LLM-based module and learnable prompts, addressing input length and semantic understanding limitations.

Findings

Improved image-text alignment in radiology tasks

Enhanced downstream performance on medical benchmarks

Effective handling of long clinical texts

Abstract

Contrastive Language-Image Pretraining (CLIP) has demonstrated strong generalization for vision-language tasks in computer vision and medical domains, yet its text encoder accepts only up to 77 tokens, which limits its ability to represent long and information-rich radiology reports. Recent adaptations using domain-specific encoders, such as PubMedBERT or ClinicalBERT, mitigate this issue by leveraging medical corpora, but remain constrained by their limited input length (typically 512 tokens) and relatively shallow semantic understanding. To address these limitations, we propose QwenCLIP, a vision-language framework that replaces CLIP's text encoder with a large language model (LLM)-based embedding module (e.g., Qwen3-Embedding) and introduces learnable prompts to enhance cross-modal alignment. By leveraging the extended context window and richer representations of LLMs, QwenCLIP captures comprehensive medical semantics from long-form clinical text, substantially improving medical image-text alignment and downstream performance on radiology benchmarks. Our code is publicly available at https://github.com/Wxy-24/QwenCLIP.