LingLanMiDian: Systematic Evaluation of LLMs on TCM Knowledge and Clinical Reasoning

TL;DR

The paper introduces LingLanMiDian, a comprehensive benchmark for evaluating large language models on Traditional Chinese Medicine tasks, highlighting current models' limitations in domain-specific reasoning and knowledge understanding.

Contribution

It presents a unified, expert-curated evaluation suite for TCM LLMs, including new metrics, protocols, and a hard subset for rigorous assessment.

Findings

Current LLMs lag behind human experts in TCM reasoning.

LingLan benchmark reveals significant gaps in knowledge recall and reasoning.

Evaluation data and code are publicly available for further research.

Abstract

Large language models (LLMs) are advancing rapidly in medical NLP, yet Traditional Chinese Medicine (TCM) with its distinctive ontology, terminology, and reasoning patterns requires domain-faithful evaluation. Existing TCM benchmarks are fragmented in coverage and scale and rely on non-unified or generation-heavy scoring that hinders fair comparison. We present the LingLanMiDian (LingLan) benchmark, a large-scale, expert-curated, multi-task suite that unifies evaluation across knowledge recall, multi-hop reasoning, information extraction, and real-world clinical decision-making. LingLan introduces a consistent metric design, a synonym-tolerant protocol for clinical labels, a per-dataset 400-item Hard subset, and a reframing of diagnosis and treatment recommendation into single-choice decision recognition. We conduct comprehensive, zero-shot evaluations on 14 leading open-source and proprietary LLMs, providing a unified perspective on their strengths and limitations in TCM commonsense knowledge understanding, reasoning, and clinical decision support; critically, the evaluation on Hard subset reveals a substantial gap between current models and human experts in TCM-specialized reasoning. By bridging fundamental knowledge and applied reasoning through standardized evaluation, LingLan establishes a unified, quantitative, and extensible foundation for advancing TCM LLMs and domain-specific medical AI research. All evaluation data and code are available at https://github.com/TCMAI-BJTU/LingLan and http://tcmnlp.com.