Human-Level and Beyond: Benchmarking Large Language Models Against Clinical Pharmacists in Prescription Review

TL;DR

This paper introduces RxBench, a comprehensive benchmark for evaluating large language models in clinical prescription review, demonstrating that some models can match or surpass human pharmacists in accuracy and robustness.

Contribution

The paper presents RxBench, a standardized, error-type-oriented benchmark for assessing LLMs in prescription review, and shows how targeted fine-tuning can improve model performance.

Findings

Leading LLMs outperform others in accuracy and robustness.

Some LLMs match or exceed licensed pharmacists in specific tasks.

Fine-tuning enhances model performance on short-answer questions.

Abstract

The rapid advancement of large language models (LLMs) has accelerated their integration into clinical decision support, particularly in prescription review. To enable systematic and fine-grained evaluation, we developed RxBench, a comprehensive benchmark that covers common prescription review categories and consolidates 14 frequent types of prescription errors drawn from authoritative pharmacy references. RxBench consists of 1,150 single-choice, 230 multiple-choice, and 879 short-answer items, all reviewed by experienced clinical pharmacists. We benchmarked 18 state-of-the-art LLMs and identified clear stratification of performance across tasks. Notably, Gemini-2.5-pro-preview-05-06, Grok-4-0709, and DeepSeek-R1-0528 consistently formed the first tier, outperforming other models in both accuracy and robustness. Comparisons with licensed pharmacists indicated that leading LLMs can match or exceed human performance in certain tasks. Furthermore, building on insights from our benchmark evaluation, we performed targeted fine-tuning on a mid-tier model, resulting in a specialized model that rivals leading general-purpose LLMs in performance on short-answer question tasks. The main contribution of RxBench lies in establishing a standardized, error-type-oriented framework that not only reveals the capabilities and limitations of frontier LLMs in prescription review but also provides a foundational resource for building more reliable and specialized clinical tools.