Learning from Medical Entity Trees: An Entity-Centric Medical Data Engineering Framework for MLLMs

TL;DR

This paper introduces an entity-centric framework that constructs a hierarchical Medical Entity Tree from literature, enhancing multimodal large language models' ability to perform detailed medical reasoning and recognition.

Contribution

It presents a novel Medical Entity Tree and a data engine that improves data curation and reasoning capabilities of MLLMs in medical applications.

Findings

Achieved state-of-the-art performance on six medical benchmarks.

Enhanced models' ability to handle complex clinical queries.

Improved fine-grained medical recognition and reasoning.

Abstract

Multimodal Large Language Models (MLLMs) have shown transformative potential in medical applications, yet their performance is hindered by conventional data curation strategies that rely on coarse-grained partitioning by modality or department. Such fragmented approaches fail to capture the hierarchical and interconnected nature of clinical medical knowledge, limiting the models' ability to perform fine-grained recognition and complex reasoning. In this paper, we propose a novel Entity-Centric Medical Data Engineering framework. We automatically extract entities from authoritative medical literature to construct a Medical Entity Tree (MET), a hierarchical structure that systematically encodes diseases, anatomical structures, modalities, and symptoms into a unified knowledge repository. Building upon the MET, we propose an advanced data engine that includes: (1) node-guided retrieval to anchor raw data to specific medical concepts, (2) a two-stage hybrid filtering and alignment pipeline to ensure precise visual-semantic correspondence, and (3) knowledge-aware data synthesis to generate enriched captions and targeted reasoning VQA pairs, leveraging structural constraints. Extensive evaluations across six medical benchmarks demonstrate that our approach significantly enhances the medical capabilities of general-purpose MLLMs, improving their ability to handle complex clinical queries and achieve state-of-the-art performance in diverse medical contexts.