Exploring the Comprehension of ChatGPT in Traditional Chinese Medicine Knowledge

TL;DR

This study evaluates ChatGPT's ability to understand and reason within Traditional Chinese Medicine using a specialized dataset, revealing strengths in true/false questions and the impact of prompt language.

Contribution

First comprehensive assessment of LLM performance in TCM domain, introducing TCM-QA dataset and analyzing prompt language effects on ChatGPT's accuracy.

Findings

ChatGPT achieves highest precision in true/false questions (0.688).

Chinese prompts outperform English prompts in evaluations.

Explanations generated by ChatGPT can enhance TCM knowledge comprehension.

Abstract

No previous work has studied the performance of Large Language Models (LLMs) in the context of Traditional Chinese Medicine (TCM), an essential and distinct branch of medical knowledge with a rich history. To bridge this gap, we present a TCM question dataset named TCM-QA, which comprises three question types: single choice, multiple choice, and true or false, to examine the LLM's capacity for knowledge recall and comprehensive reasoning within the TCM domain. In our study, we evaluate two settings of the LLM, zero-shot and few-shot settings, while concurrently discussing the differences between English and Chinese prompts. Our results indicate that ChatGPT performs best in true or false questions, achieving the highest precision of 0.688 while scoring the lowest precision is 0.241 in multiple-choice questions. Furthermore, we observed that Chinese prompts outperformed English prompts in our evaluations. Additionally, we assess the quality of explanations generated by ChatGPT and their potential contribution to TCM knowledge comprehension. This paper offers valuable insights into the applicability of LLMs in specialized domains and paves the way for future research in leveraging these powerful models to advance TCM.