# Retrieval Augmented Generation (RAG) for Evaluating Regulatory Compliance of Drug Information and Clinical Trial Protocols

## TL;DR

This study shows that AI systems using retrieval-augmented generation can help check if drug information and clinical trial protocols follow regulatory guidelines, making the process faster and more efficient.

## Contribution

The study demonstrates that RAG-based AI improves LLMs' ability to evaluate regulatory compliance in clinical pharmacology documents.

## Key findings

- RAG systems correctly identified drug indications, population use, and warnings for several medications.
- The system explained the reasons for the withdrawal of certain drugs like rofecoxib and valdecoxib.
- RAG-based evaluations aligned with manual reviews of clinical trial protocols.

## Abstract

The purpose was to evaluate retrieval‐augmented generative (RAG) artificial intelligence (AI) methods for assessing the regulatory compliance of drug information and adherence to best practices in clinical trial protocols. Integrated systems containing RAG and large language model (LLM) components were employed to evaluate drug information and clinical trial protocols. The drug information for adalimumab, insulin glargine, atorvastatin calcium, sertraline, and alprazolam was evaluated for compliance with Food and Drug Administration (FDA) clinical pharmacology guidance for indications, use in specific populations, and warnings and precautions. The reasons for the withdrawal of rofecoxib, valdecoxib, and troglitazone were elicited. The clinical trial protocol evaluation system was used to assess a Phase‐2a clinical trial protocol of Rifafour in tuberculosis with the FDA E9 and E9 (R1) guidance documents. The RAG system correctly identified the indications, use in specific populations, and warnings and precautions for adalimumab, insulin glargine, atorvastatin calcium, sertraline, and alprazolam. The drug information was evaluated against the requirements in the guidance documents, confirming compliance when present and providing explanations for deficiencies. The causes underlying the withdrawal of rofecoxib, valdecoxib, and troglitazone were explained. The clinical protocol summary included study design, population definitions, treatments, dose levels, and route of administration. The summary of the statistical analysis plan included primary/secondary endpoints, statistical tests, pharmacokinetic parameters, and handling of missing data and outliers. The findings aligned with manual protocol reviews. RAG‐based AI methods can improve the usefulness of LLMs in document‐restricted settings and are a promising approach for evaluating the compliance of clinical pharmacology documents.

What is the current knowledge on the topic?
○The technical complexities involved in evaluating drug information and reviewing clinical protocols require multidisciplinary expertise to assess scientific validity, risks, and opportunities, making the process time‐consuming and expertise‐intensive. Large language models (LLMs) offer a transformative opportunity to reshape this workflow.
What question did this study address?
○The study evaluated retrieval‐augmented generative (RAG) AI methods for assessing the regulatory compliance of drug information and examining adherence to best practices in clinical trial protocols.
What does this study add to our knowledge?
○The results show that RAG systems improve the utility of LLMs and are a promising approach for diverse clinical pharmacology challenges.
How might this change drug discovery, development, and/or therapeutics?
○RAG‐based systems may help sponsors systematically review drug documents for compliance gaps before regulatory submission.

What is the current knowledge on the topic?
○The technical complexities involved in evaluating drug information and reviewing clinical protocols require multidisciplinary expertise to assess scientific validity, risks, and opportunities, making the process time‐consuming and expertise‐intensive. Large language models (LLMs) offer a transformative opportunity to reshape this workflow.

The technical complexities involved in evaluating drug information and reviewing clinical protocols require multidisciplinary expertise to assess scientific validity, risks, and opportunities, making the process time‐consuming and expertise‐intensive. Large language models (LLMs) offer a transformative opportunity to reshape this workflow.

What question did this study address?
○The study evaluated retrieval‐augmented generative (RAG) AI methods for assessing the regulatory compliance of drug information and examining adherence to best practices in clinical trial protocols.

The study evaluated retrieval‐augmented generative (RAG) AI methods for assessing the regulatory compliance of drug information and examining adherence to best practices in clinical trial protocols.

What does this study add to our knowledge?
○The results show that RAG systems improve the utility of LLMs and are a promising approach for diverse clinical pharmacology challenges.

The results show that RAG systems improve the utility of LLMs and are a promising approach for diverse clinical pharmacology challenges.

How might this change drug discovery, development, and/or therapeutics?
○RAG‐based systems may help sponsors systematically review drug documents for compliance gaps before regulatory submission.

RAG‐based systems may help sponsors systematically review drug documents for compliance gaps before regulatory submission.