Assessing LLMs' Performance: Insights from the Chinese Pharmacist Exam

TL;DR

This study compares the performance of two large language models, ChatGPT-4o and DeepSeek-R1, on the Chinese Pharmacist Licensing Examination, revealing that the domain-specific DeepSeek-R1 outperforms the general-purpose ChatGPT-4o in accuracy and consistency.

Contribution

The paper provides a comparative analysis of LLMs on real exam questions, highlighting the superior performance of a domain-specific model in a high-stakes certification context.

Findings

DeepSeek-R1 achieved 90.0% accuracy versus 76.1% for ChatGPT-4o.

DeepSeek-R1 showed consistent advantages across modules.

Performance differences were not statistically significant in year-wise comparisons.

Abstract

Background: As large language models (LLMs) become increasingly integrated into digital health education and assessment workflows, their capabilities in supporting high-stakes, domain-specific certification tasks remain underexplored.In China, the national pharmacist licensure exam serves as a standardized benchmark for evaluating pharmacists' clinical and theoretical competencies. Objective: This study aimed to compare the performance of two LLMs: ChatGPT-4o and DeepSeek-R1 on real questions from the Chinese Pharmacist Licensing Examination (2017-2021), and to discuss the implications of these performance differences for AI-enabled formative evaluation. Methods: A total of 2,306 multiple-choice (text-only) questions were compiled from official exams, training materials, and public databases. Questions containing tables or images were excluded. Each item was input in its original Chinese format, and model responses were evaluated for exact accuracy. Pearson's Chi-squared test was used to compare overall performance, and Fisher's exact test was applied to year-wise multiple-choice accuracy. Results: DeepSeek-R1 outperformed ChatGPT-4o with a significantly higher overall accuracy (90.0% vs. 76.1%, p < 0.001). Unit-level analyses revealed consistent advantages for DeepSeek-R1, particularly in foundational and clinical synthesis modules. While year-by-year multiple-choice performance also favored DeepSeek-R1, this performance gap did not reach statistical significance in any specific unit-year (all p > 0.05). Conclusion: DeepSeek-R1 demonstrated robust alignment with the structural and semantic demands of the pharmacist licensure exam. These findings suggest that domain-specific models warrant further investigation for this context, while also reinforcing the necessity of human oversight in legally and ethically sensitive contexts.