A Smart Multimodal Healthcare Copilot with Powerful LLM Reasoning
Xuejiao Zhao, Siyan Liu, Su-Yin Yang, Chunyan Miao

TL;DR
MedRAG is a multimodal healthcare assistant leveraging large language models and knowledge graphs to improve diagnostic accuracy and decision-making, reducing misdiagnosis risks in medical settings.
Contribution
It introduces MedRAG, a novel multimodal healthcare copilot that combines LLM reasoning with knowledge graph-augmented retrieval for enhanced medical diagnosis.
Findings
Outperforms existing models on public and private datasets
Provides more specific and accurate healthcare recommendations
Supports multiple input modalities including voice and EHRs
Abstract
Misdiagnosis causes significant harm to healthcare systems worldwide, leading to increased costs and patient risks. MedRAG is a smart multimodal healthcare copilot equipped with powerful large language model (LLM) reasoning, designed to enhance medical decision-making. It supports multiple input modalities, including non-intrusive voice monitoring, general medical queries, and electronic health records. MedRAG provides recommendations on diagnosis, treatment, medication, and follow-up questioning. Leveraging retrieval-augmented generation enhanced by knowledge graph-elicited reasoning, MedRAG retrieves and integrates critical diagnostic insights, reducing the risk of misdiagnosis. It has been evaluated on both public and private datasets, outperforming existing models and offering more specific and accurate healthcare assistance. A demonstration video of MedRAG is available at:…
Peer Reviews
No public reviews on file for this paper yet. If you reviewed it on a platform where reviews are public (OpenReview, ICLR, NeurIPS, ICML), you can paste yours below so the community can read it here.
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsSemantic Web and Ontologies · Service-Oriented Architecture and Web Services
