A benchmark dataset for evaluating Syndrome Differentiation and Treatment in large language models

TL;DR

This paper introduces TCM-BEST4SDT, a comprehensive benchmark dataset for evaluating large language models' capabilities in Traditional Chinese Medicine, focusing on syndrome differentiation and treatment decision-making.

Contribution

It presents a novel, expert-annotated benchmark with multiple evaluation mechanisms specifically designed for TCM applications of LLMs.

Findings

Effective evaluation of 15 mainstream LLMs on TCM tasks.

Benchmark covers knowledge, ethics, safety, and syndrome differentiation.

Public release of the TCM-BEST4SDT dataset for future research.

Abstract

The emergence of Large Language Models (LLMs) within the Traditional Chinese Medicine (TCM) domain presents an urgent need to assess their clinical application capabilities. However, such evaluations are challenged by the individualized, holistic, and diverse nature of TCM's "Syndrome Differentiation and Treatment" (SDT). Existing benchmarks are confined to knowledge-based question-answering or the accuracy of syndrome differentiation, often neglecting assessment of treatment decision-making. Here, we propose a comprehensive, clinical case-based benchmark spearheaded by TCM experts, and a specialized reward model employed to quantify prescription-syndrome congruence. Data annotation follows a rigorous pipeline. This benchmark, designated TCM-BEST4SDT, encompasses four tasks, including TCM Basic Knowledge, Medical Ethics, LLM Content Safety, and SDT. The evaluation framework integrates three mechanisms, namely selected-response evaluation, judge model evaluation, and reward model evaluation. The effectiveness of TCM-BEST4SDT was corroborated through experiments on 15 mainstream LLMs, spanning both general and TCM domains. To foster the development of intelligent TCM research, TCM-BEST4SDT is now publicly available.