A Medical Multimodal Diagnostic Framework Integrating Vision-Language Models and Logic Tree Reasoning

TL;DR

This paper introduces a multimodal diagnostic framework that combines vision-language models with logic-based reasoning to improve accuracy and interpretability in medical AI diagnostics.

Contribution

It presents a novel framework integrating vision-language alignment with logic tree reasoning, enhancing trustworthiness and interpretability in multimodal medical diagnosis.

Findings

Improved diagnostic accuracy on MedXpertQA benchmark

More interpretable reasoning traces produced

Competitive performance on text-only tasks

Abstract

With the rapid growth of large language models (LLMs) and vision-language models (VLMs) in medicine, simply integrating clinical text and medical imaging does not guarantee reliable reasoning. Existing multimodal models often produce hallucinations or inconsistent chains of thought, limiting clinical trust. We propose a diagnostic framework built upon LLaVA that combines vision-language alignment with logic-regularized reasoning. The system includes an input encoder for text and images, a projection module for cross-modal alignment, a reasoning controller that decomposes diagnostic tasks into steps, and a logic tree generator that assembles stepwise premises into verifiable conclusions. Evaluations on MedXpertQA and other benchmarks show that our method improves diagnostic accuracy and yields more interpretable reasoning traces on multimodal tasks, while remaining competitive on text-only settings. These results suggest a promising step toward trustworthy multimodal medical AI.