Performance of large language models in numerical vs. semantic medical knowledge: Benchmarking on evidence-based Q&As

TL;DR

This study benchmarks large language models' ability to answer evidence-based medical questions, revealing they perform better on semantic than numerical questions but still lag behind human experts, highlighting limitations in clinical decision support.

Contribution

The paper introduces EBMQA, a large-scale medical question dataset, and compares LLM performance on numerical and semantic questions, highlighting their strengths and limitations in clinical knowledge application.

Findings

LLMs perform better on semantic questions than numerical ones.

Claude3 outperforms GPT-4 in numerical question accuracy.

Both LLMs are inferior to human experts in medical question answering.

Abstract

Clinical problem-solving requires processing of semantic medical knowledge such as illness scripts and numerical medical knowledge of diagnostic tests for evidence-based decision-making. As large language models (LLMs) show promising results in many aspects of language-based clinical practice, their ability to generate non-language evidence-based answers to clinical questions is inherently limited by tokenization. Therefore, we evaluated LLMs' performance on two question types: numeric (correlating findings) and semantic (differentiating entities) while examining differences within and between LLMs in medical aspects and comparing their performance to humans. To generate straightforward multi-choice questions and answers (QAs) based on evidence-based medicine (EBM), we used a comprehensive medical knowledge graph (encompassed data from more than 50,00 peer-reviewed articles) and created the "EBMQA". EBMQA contains 105,000 QAs labeled with medical and non-medical topics and classified into numerical or semantic questions. We benchmarked this dataset using more than 24,500 QAs on two state-of-the-art LLMs: Chat-GPT4 and Claude3-Opus. We evaluated the LLMs accuracy on semantic and numerical question types and according to sub-labeled topics. For validation, six medical experts were tested on 100 numerical EBMQA questions. We found that both LLMs excelled more in semantic than numerical QAs, with Claude3 surpassing GPT4 in numerical QAs. However, both LLMs showed inter and intra gaps in different medical aspects and remained inferior to humans. Thus, their medical advice should be addressed carefully.