Querying GI Endoscopy Images: A VQA Approach

TL;DR

This paper explores adapting a multimodal language model to answer visual questions about gastrointestinal endoscopy images, aiming to improve diagnostic AI tools in medical imaging.

Contribution

It evaluates the Florence2 model's adaptation for GI endoscopy VQA tasks and assesses its performance with standard NLP metrics.

Findings

Model shows potential in medical VQA tasks

Standard metrics used for evaluation

Highlights challenges in domain-specific VQA

Abstract

VQA (Visual Question Answering) combines Natural Language Processing (NLP) with image understanding to answer questions about a given image. It has enormous potential for the development of medical diagnostic AI systems. Such a system can help clinicians diagnose gastro-intestinal (GI) diseases accurately and efficiently. Although many of the multimodal LLMs available today have excellent VQA capabilities in the general domain, they perform very poorly for VQA tasks in specialized domains such as medical imaging. This study is a submission for ImageCLEFmed-MEDVQA-GI 2025 subtask 1 that explores the adaptation of the Florence2 model to answer medical visual questions on GI endoscopy images. We also evaluate the model performance using standard metrics like ROUGE, BLEU and METEOR