Clinical trial cohort selection using Large Language Models on n2c2 Challenges

TL;DR

This paper evaluates the effectiveness of large language models in automating clinical trial cohort selection, highlighting their strengths in simple tasks and challenges in complex reasoning scenarios.

Contribution

It benchmarks LLM performance on clinical trial cohort selection using n2c2 challenges, revealing their potential and limitations in this domain.

Findings

LLMs perform well on simple cohort selection tasks.

Challenges arise when fine-grained knowledge and reasoning are required.

The study provides insights into LLM capabilities and limitations in clinical NLP.

Abstract

Clinical trials are a critical process in the medical field for introducing new treatments and innovations. However, cohort selection for clinical trials is a time-consuming process that often requires manual review of patient text records for specific keywords. Though there have been studies on standardizing the information across the various platforms, Natural Language Processing (NLP) tools remain crucial for spotting eligibility criteria in textual reports. Recently, pre-trained large language models (LLMs) have gained popularity for various NLP tasks due to their ability to acquire a nuanced understanding of text. In this paper, we study the performance of large language models on clinical trial cohort selection and leverage the n2c2 challenges to benchmark their performance. Our results are promising with regard to the incorporation of LLMs for simple cohort selection tasks, but also highlight the difficulties encountered by these models as soon as fine-grained knowledge and reasoning are required.