Targeted Visual Prompting for Medical Visual Question Answering

TL;DR

This paper introduces targeted visual prompting to improve medical visual question answering by enabling multimodal large language models to better understand specific image regions, enhancing their interpretative capabilities.

Contribution

It proposes a novel targeted visual prompting method that incorporates region-based questions into MLLMs, improving visual understanding in Med-VQA tasks.

Findings

Effective across multiple datasets

Outperforms baseline models

Enhances region-based visual understanding

Abstract

With growing interest in recent years, medical visual question answering (Med-VQA) has rapidly evolved, with multimodal large language models (MLLMs) emerging as an alternative to classical model architectures. Specifically, their ability to add visual information to the input of pre-trained LLMs brings new capabilities for image interpretation. However, simple visual errors cast doubt on the actual visual understanding abilities of these models. To address this, region-based questions have been proposed as a means to assess and enhance actual visual understanding through compositional evaluation. To combine these two perspectives, this paper introduces targeted visual prompting to equip MLLMs with region-based questioning capabilities. By presenting the model with both the isolated region and the region in its context in a customized visual prompt, we show the effectiveness of our method across multiple datasets while comparing it to several baseline models. Our code and data are available at https://github.com/sergiotasconmorales/locvqallm.