Visual Prompt Engineering for Vision Language Models in Radiology

TL;DR

This paper introduces a method to improve zero-shot radiology image classification by embedding visual cues like arrows and bounding boxes into images, which guides models to focus on relevant regions and enhances interpretability and accuracy.

Contribution

The study demonstrates that incorporating visual markers into radiological images significantly improves classification performance and interpretability in zero-shot settings.

Findings

Visual markers increase AUROC by up to 0.185.

Attention maps show models focus on clinically relevant areas.

Method is validated across four public chest X-ray datasets.

Abstract

Medical image classification plays a crucial role in clinical decision-making, yet most models are constrained to a fixed set of predefined classes, limiting their adaptability to new conditions. Contrastive Language-Image Pretraining (CLIP) offers a promising solution by enabling zero-shot classification through multimodal large-scale pretraining. However, while CLIP effectively captures global image content, radiology requires a more localized focus on specific pathology regions to enhance both interpretability and diagnostic accuracy. To address this, we explore the potential of incorporating visual cues into zero-shot classification, embedding visual markers, such as arrows, bounding boxes, and circles, directly into radiological images to guide model attention. Evaluating across four public chest X-ray datasets, we demonstrate that visual markers improve AUROC by up to 0.185, highlighting their effectiveness in enhancing classification performance. Furthermore, attention map analysis confirms that visual cues help models focus on clinically relevant areas, leading to more interpretable predictions.To support further research, we use public datasets and provide our codebase and preprocessing pipeline under https://github.com/MIC-DKFZ/VPE-in-Radiology, serving as a reference point for future work on localized classification in medical imaging.