MedEthicsQA: A Comprehensive Question Answering Benchmark for Medical Ethics Evaluation of LLMs

TL;DR

MedEthicsQA is a new comprehensive benchmark with thousands of questions designed to evaluate the ethical reasoning of medical large language models, revealing their current limitations in medical ethics understanding.

Contribution

This paper introduces MedEthicsQA, the first large-scale, hierarchically structured benchmark for assessing medical ethics in LLMs, combining multiple datasets and expert validation.

Findings

State-of-the-art MedLLMs perform poorly on ethics questions.

The benchmark reveals significant gaps in medical ethics alignment.

High-quality, validated dataset with low error rate.

Abstract

While Medical Large Language Models (MedLLMs) have demonstrated remarkable potential in clinical tasks, their ethical safety remains insufficiently explored. This paper introduces $\textbf{MedEthicsQA}$, a comprehensive benchmark comprising $\textbf{5,623}$ multiple-choice questions and $\textbf{5,351}$ open-ended questions for evaluation of medical ethics in LLMs. We systematically establish a hierarchical taxonomy integrating global medical ethical standards. The benchmark encompasses widely used medical datasets, authoritative question banks, and scenarios derived from PubMed literature. Rigorous quality control involving multi-stage filtering and multi-faceted expert validation ensures the reliability of the dataset with a low error rate ($2.72\%$). Evaluation of state-of-the-art MedLLMs exhibit declined performance in answering medical ethics questions compared to their foundation counterparts, elucidating the deficiencies of medical ethics alignment. The dataset, registered under CC BY-NC 4.0 license, is available at https://github.com/JianhuiWei7/MedEthicsQA.