ShizhenGPT: Towards Multimodal LLMs for Traditional Chinese Medicine

TL;DR

ShizhenGPT is a pioneering multimodal large language model designed specifically for Traditional Chinese Medicine, integrating diverse sensory data to enhance diagnostic reasoning and understanding.

Contribution

It introduces the first multimodal LLM for TCM, curates the largest TCM dataset, and demonstrates superior multimodal reasoning and visual understanding capabilities.

Findings

Outperforms comparable-scale LLMs in TCM tasks

Leads in visual understanding among multimodal LLMs

Achieves unified perception across multiple sensory modalities

Abstract

Despite the success of large language models (LLMs) in various domains, their potential in Traditional Chinese Medicine (TCM) remains largely underexplored due to two critical barriers: (1) the scarcity of high-quality TCM data and (2) the inherently multimodal nature of TCM diagnostics, which involve looking, listening, smelling, and pulse-taking. These sensory-rich modalities are beyond the scope of conventional LLMs. To address these challenges, we present ShizhenGPT, the first multimodal LLM tailored for TCM. To overcome data scarcity, we curate the largest TCM dataset to date, comprising 100GB+ of text and 200GB+ of multimodal data, including 1.2M images, 200 hours of audio, and physiological signals. ShizhenGPT is pretrained and instruction-tuned to achieve deep TCM knowledge and multimodal reasoning. For evaluation, we collect recent national TCM qualification exams and build a visual benchmark for Medicinal Recognition and Visual Diagnosis. Experiments demonstrate that ShizhenGPT outperforms comparable-scale LLMs and competes with larger proprietary models. Moreover, it leads in TCM visual understanding among existing multimodal LLMs and demonstrates unified perception across modalities like sound, pulse, smell, and vision, paving the way toward holistic multimodal perception and diagnosis in TCM. Datasets, models, and code are publicly available. We hope this work will inspire further exploration in this field.