PRISM: Patient Records Interpretation for Semantic Clinical Trial Matching using Large Language Models

TL;DR

This paper evaluates the effectiveness of large language models, including GPT-4, GPT-3.5, and a custom model OncoLLM, in accurately matching real-world patient records to suitable clinical trials, demonstrating that smaller models can outperform larger ones and match medical experts.

Contribution

First comprehensive empirical study of LLMs for clinical trial matching using real-world EHRs, showing competitive performance with medical doctors.

Findings

OncoLLM outperforms GPT-3.5 in matching accuracy.

OncoLLM matches the performance of qualified medical doctors.

Real-world EHRs used for evaluation demonstrate practical applicability.

Abstract

Clinical trial matching is the task of identifying trials for which patients may be potentially eligible. Typically, this task is labor-intensive and requires detailed verification of patient electronic health records (EHRs) against the stringent inclusion and exclusion criteria of clinical trials. This process is manual, time-intensive, and challenging to scale up, resulting in many patients missing out on potential therapeutic options. Recent advancements in Large Language Models (LLMs) have made automating patient-trial matching possible, as shown in multiple concurrent research studies. However, the current approaches are confined to constrained, often synthetic datasets that do not adequately mirror the complexities encountered in real-world medical data. In this study, we present the first, end-to-end large-scale empirical evaluation of clinical trial matching using real-world EHRs. Our study showcases the capability of LLMs to accurately match patients with appropriate clinical trials. We perform experiments with proprietary LLMs, including GPT-4 and GPT-3.5, as well as our custom fine-tuned model called OncoLLM and show that OncoLLM, despite its significantly smaller size, not only outperforms GPT-3.5 but also matches the performance of qualified medical doctors. All experiments were carried out on real-world EHRs that include clinical notes and available clinical trials from a single cancer center in the United States.