47B Mixture-of-Experts Beats 671B Dense Models on Chinese Medical Examinations

TL;DR

This paper evaluates 27 large language models on Chinese medical exam questions across multiple specialties, revealing performance variations and highlighting the potential and limitations of LLMs in medical applications.

Contribution

It introduces a comprehensive benchmark framework and provides empirical insights into model performance across specialties and difficulty levels in Chinese medical exams.

Findings

Mixtral-8x7B achieves 74.25% accuracy, outperforming larger models.

Smaller mixture-of-experts models perform competitively with larger dense models.

Models show consistent performance across different physician difficulty levels.

Abstract

The rapid advancement of large language models(LLMs) has prompted significant interest in their potential applications in medical domains. This paper presents a comprehensive benchmark evaluation of 27 state-of-the-art LLMs on Chinese medical examination questions, encompassing seven medical specialties across two professional levels. We introduce a robust evaluation framework that assesses model performance on 2,800 carefully curated questions from cardiovascular, gastroenterology, hematology, infectious diseases, nephrology, neurology, and respiratory medicine domains. Our dataset distinguishes between attending physician and senior physician difficulty levels, providing nuanced insights into model capabilities across varying complexity. Our empirical analysis reveals substantial performance variations among models, with Mixtral-8x7B achieving the highest overall accuracy of 74.25%, followed by DeepSeek-R1-671B at 64.07%. Notably, we observe no consistent correlation between model size and performance, as evidenced by the strong performance of smaller mixture-of-experts architectures. The evaluation demonstrates significant performance gaps between medical specialties, with models generally performing better on cardiovascular and neurology questions compared to gastroenterology and nephrology domains. Furthermore, our analysis indicates minimal performance degradation between attending and senior physician levels for top-performing models, suggesting robust generalization capabilities. This benchmark provides critical insights for the deployment of LLMs in medical education and clinical decision support systems, highlighting both the promise and current limitations of these technologies in specialized medical contexts.