MedVQA-TREE: A Multimodal Reasoning and Retrieval Framework for Sarcopenia Prediction

TL;DR

MedVQA-TREE is a multimodal AI framework that combines hierarchical image analysis, feature fusion, and knowledge retrieval to improve sarcopenia diagnosis from ultrasound images, achieving high accuracy and surpassing previous methods.

Contribution

It introduces a novel multimodal framework integrating hierarchical image interpretation, gated feature fusion, and multi-hop knowledge retrieval for sarcopenia prediction.

Findings

Achieved up to 99% diagnostic accuracy.

Outperformed previous state-of-the-art by over 10%.

Validated on multiple datasets including a custom sarcopenia ultrasound dataset.

Abstract

Accurate sarcopenia diagnosis via ultrasound remains challenging due to subtle imaging cues, limited labeled data, and the absence of clinical context in most models. We propose MedVQA-TREE, a multimodal framework that integrates a hierarchical image interpretation module, a gated feature-level fusion mechanism, and a novel multi-hop, multi-query retrieval strategy. The vision module includes anatomical classification, region segmentation, and graph-based spatial reasoning to capture coarse, mid-level, and fine-grained structures. A gated fusion mechanism selectively integrates visual features with textual queries, while clinical knowledge is retrieved through a UMLS-guided pipeline accessing PubMed and a sarcopenia-specific external knowledge base. MedVQA-TREE was trained and evaluated on two public MedVQA datasets (VQA-RAD and PathVQA) and a custom sarcopenia ultrasound dataset. The model achieved up to 99% diagnostic accuracy and outperformed previous state-of-the-art methods by over 10%. These results underscore the benefit of combining structured visual understanding with guided knowledge retrieval for effective AI-assisted diagnosis in sarcopenia.