Guiding Medical Vision-Language Models with Explicit Visual Prompts: Framework Design and Comprehensive Exploration of Prompt Variations

TL;DR

This paper introduces MedVP, a framework that uses explicit visual prompts to guide medical vision-language models, improving their focus on specific regions and enhancing performance in medical visual question answering tasks.

Contribution

The paper presents a novel MedVP framework that integrates visual prompts with medical entity extraction and dataset adaptation for improved model focus and accuracy.

Findings

Outperforms recent state-of-the-art models on multiple medical VQA datasets.

Visual prompts significantly improve region-specific attention and model performance.

Extensive experiments validate the effectiveness and clinical relevance of the approach.

Abstract

While mainstream vision-language models (VLMs) have advanced rapidly in understanding image level information, they still lack the ability to focus on specific areas designated by humans. Rather, they typically rely on large volumes of high-quality image-text paired data to learn and generate posterior attention maps. To address this critical issue, we propose leveraging visual prompts:simple visual markers in various forms to guide and enhance the formation of region-specific attention. Thus, we introduce MedVP, a pioneering framework that integrates medical entity extraction, visual prompt generation, and dataset adaptation for visual prompt guided fine-tuning. We successfully outperform recent state-of-the-art large models across multiple medical VQA datasets. Extensive experiments and Human evaluation are conducted to analyze the impact of different visual prompt forms and how they contribute to performance improvement. The results demonstrate both the effectiveness and clinical significance of our approach.