RxSafeBench: Identifying Medication Safety Issues of Large Language Models in Simulated Consultation

TL;DR

This paper introduces RxSafeBench, a comprehensive benchmark for evaluating medication safety in large language models within simulated clinical consultations, addressing a critical gap in healthcare AI safety assessment.

Contribution

It creates a realistic, high-quality benchmark with a large safety database and evaluates LLMs' ability to recommend safe medications, highlighting current limitations.

Findings

LLMs struggle with contraindication and interaction knowledge.

Risks are harder to detect when implied rather than explicit.

Benchmark enables systematic assessment of medication safety in LLMs.

Abstract

Numerous medical systems powered by Large Language Models (LLMs) have achieved remarkable progress in diverse healthcare tasks. However, research on their medication safety remains limited due to the lack of real world datasets, constrained by privacy and accessibility issues. Moreover, evaluation of LLMs in realistic clinical consultation settings, particularly regarding medication safety, is still underexplored. To address these gaps, we propose a framework that simulates and evaluates clinical consultations to systematically assess the medication safety capabilities of LLMs. Within this framework, we generate inquiry diagnosis dialogues with embedded medication risks and construct a dedicated medication safety database, RxRisk DB, containing 6,725 contraindications, 28,781 drug interactions, and 14,906 indication-drug pairs. A two-stage filtering strategy ensures clinical realism and professional quality, resulting in the benchmark RxSafeBench with 2,443 high-quality consultation scenarios. We evaluate leading open-source and proprietary LLMs using structured multiple choice questions that test their ability to recommend safe medications under simulated patient contexts. Results show that current LLMs struggle to integrate contraindication and interaction knowledge, especially when risks are implied rather than explicit. Our findings highlight key challenges in ensuring medication safety in LLM-based systems and provide insights into improving reliability through better prompting and task-specific tuning. RxSafeBench offers the first comprehensive benchmark for evaluating medication safety in LLMs, advancing safer and more trustworthy AI-driven clinical decision support.