ACR: A Benchmark for Automatic Cohort Retrieval

TL;DR

This paper introduces ACR, a new benchmark task for automatic patient cohort retrieval using large language models and neuro-symbolic methods, addressing challenges in longitudinal reasoning and data complexity in healthcare.

Contribution

It presents a comprehensive benchmark with datasets and evaluation framework for ACR, evaluating LLMs and neuro-symbolic approaches for healthcare cohort retrieval.

Findings

LLMs and neuro-symbolic methods show promise for ACR tasks

High-quality, efficient ACR systems require longitudinal reasoning capabilities

Benchmark datasets facilitate future research in automated cohort retrieval

Abstract

Identifying patient cohorts is fundamental to numerous healthcare tasks, including clinical trial recruitment and retrospective studies. Current cohort retrieval methods in healthcare organizations rely on automated queries of structured data combined with manual curation, which are time-consuming, labor-intensive, and often yield low-quality results. Recent advancements in large language models (LLMs) and information retrieval (IR) offer promising avenues to revolutionize these systems. Major challenges include managing extensive eligibility criteria and handling the longitudinal nature of unstructured Electronic Medical Records (EMRs) while ensuring that the solution remains cost-effective for real-world application. This paper introduces a new task, Automatic Cohort Retrieval (ACR), and evaluates the performance of LLMs and commercial, domain-specific neuro-symbolic approaches. We provide a benchmark task, a query dataset, an EMR dataset, and an evaluation framework. Our findings underscore the necessity for efficient, high-quality ACR systems capable of longitudinal reasoning across extensive patient databases.