Retrieval Augmented Large Language Model System for Comprehensive Drug Contraindications

TL;DR

This paper presents a Retrieval Augmented Generation system that significantly improves large language models' accuracy in providing reliable pharmaceutical contraindication information, enhancing healthcare decision support.

Contribution

It introduces a RAG pipeline using GPT-4o-mini and embedding models to effectively retrieve and re-rank drug contraindication data, improving accuracy over baseline models.

Findings

Model accuracy increased from 0.49-0.57 to over 0.87-0.94 after RAG integration.

The system effectively handles contraindications for age, pregnancy, and drug combinations.

Augmented LLMs can reduce uncertainty in prescription decisions.

Abstract

The versatility of large language models (LLMs) has been explored across various sectors, but their application in healthcare poses challenges, particularly in the domain of pharmaceutical contraindications where accurate and reliable information is required. This study enhances the capability of LLMs to address contraindications effectively by implementing a Retrieval Augmented Generation (RAG) pipeline. Utilizing OpenAI's GPT-4o-mini as the base model, and the text-embedding-3-small model for embeddings, our approach integrates Langchain to orchestrate a hybrid retrieval system with re-ranking. This system leverages Drug Utilization Review (DUR) data from public databases, focusing on contraindications for specific age groups, pregnancy, and concomitant drug use. The dataset includes 300 question-answer pairs across three categories, with baseline model accuracy ranging from 0.49 to 0.57. Post-integration of the RAG pipeline, we observed a significant improvement in model accuracy, achieving rates of 0.94, 0.87, and 0.89 for contraindications related to age groups, pregnancy, and concomitant drug use, respectively. The results indicate that augmenting LLMs with a RAG framework can substantially reduce uncertainty in prescription and drug intake decisions by providing more precise and reliable drug contraindication information.