GEMeX-RMCoT: An Enhanced Med-VQA Dataset for Region-Aware Multimodal Chain-of-Thought Reasoning

TL;DR

This paper introduces GEMeX-RMCoT, a new medical visual question answering dataset with region-aware reasoning steps and a reinforcement learning reward mechanism, enhancing interpretability and efficiency in clinical decision support models.

Contribution

It presents a novel region-aware multimodal reasoning dataset and a reinforcement learning method to improve interpretability and data efficiency in medical VQA models.

Findings

Achieves comparable performance with only 12.5% of training data

Provides fine-grained visual region grounding for explainability

Enhances model reliability and interpretability

Abstract

Medical visual question answering aims to support clinical decision-making by enabling models to answer natural language questions based on medical images. While recent advances in multi-modal learning have significantly improved performance, current methods still suffer from limited answer reliability and poor interpretability, impairing the ability of clinicians and patients to understand and trust model outputs. To address these limitations, this work first proposes a Region-Aware Multimodal Chain-of-Thought (RMCoT) dataset, in which the process of producing an answer is preceded by a sequence of intermediate reasoning steps that explicitly ground relevant visual regions of the medical image, thereby providing fine-grained explainability. Furthermore, we introduce a novel verifiable reward mechanism for reinforcement learning to guide post-training, improving the alignment between the model's reasoning process and its final answer. Remarkably, our method achieves comparable performance using only one-eighth of the training data, demonstrating the efficiency and effectiveness of the proposal. The dataset is available at https://www.med-vqa.com/GEMeX/.