DrugRAG: Enhancing Pharmacy LLM Performance Through A Novel Retrieval-Augmented Generation Pipeline

TL;DR

This paper introduces DrugRAG, a retrieval-augmented generation pipeline that enhances pharmacy question-answering accuracy of large language models by integrating external drug knowledge without altering the models.

Contribution

The study presents DrugRAG, a novel external knowledge retrieval method that significantly improves LLM performance on pharmacy tasks without changing model architecture.

Findings

DrugRAG increased accuracy by 7 to 21 percentage points across models.

Statistically significant improvements were observed mainly in smaller and mid-sized open-source models.

Benchmarking showed baseline performance ranged from 46% to 92%, with DrugRAG boosting accuracy.

Abstract

In our study, we evaluated large language model (LLM) performance on pharmacy licensure-style question-answering tasks and developed an external knowledge integration method to improve accuracy. We benchmarked ten LLMs with varying parameter sizes (8 billion to 70+ billion) using a 141-question pharmacy dataset, measuring baseline accuracy without modification. Baseline performance ranged from 46% to 92%, with GPT-5 (92%) and o3 (89%) achieving the highest scores, while smaller open-source models showed substantially lower performance. We then developed DrugRAG, a three-step retrieval-augmented generation (RAG) pipeline that retrieves structured, evidence-based drug information and augments model prompts with contextual pharmacological evidence, operating externally and requiring no changes to model architecture or parameters. DrugRAG increased accuracy across all five evaluated models, with gains ranging from 7 to 21 percentage points (e.g., Gemma 3 27B: 61.0% to 71%, Llama 3.1 8B: 46% to 67%). McNemar analyses demonstrated statistically significant paired improvements primarily in smaller and mid-sized open-source models. These findings demonstrate that integrating structured external drug knowledge via DrugRAG can improve LLM performance on pharmacy-focused question-answering tasks without modifying the underlying models, providing a practical pipeline for enhancing evidence-based pharmacy-focused AI applications.