Leveraging Large Language Models for Medical Information Extraction and Query Generation

TL;DR

This paper demonstrates that small, open-source large language models can effectively generate queries for clinical trial retrieval, matching or surpassing expert performance while being computationally efficient and practical for medical applications.

Contribution

The study evaluates multiple small, open-source LLMs for medical query generation, showing they can achieve retrieval performance comparable to experts with faster response times.

Findings

LLMs achieve retrieval effectiveness on par with or better than medical experts.

Open-source LLMs generate concise, effective queries within seconds.

Models are suitable for real-world clinical trial retrieval applications.

Abstract

This paper introduces a system that integrates large language models (LLMs) into the clinical trial retrieval process, enhancing the effectiveness of matching patients with eligible trials while maintaining information privacy and allowing expert oversight. We evaluate six LLMs for query generation, focusing on open-source and relatively small models that require minimal computational resources. Our evaluation includes two closed-source and four open-source models, with one specifically trained in the medical field and five general-purpose models. We compare the retrieval effectiveness achieved by LLM-generated queries against those created by medical experts and state-of-the-art methods from the literature. Our findings indicate that the evaluated models reach retrieval effectiveness on par with or greater than expert-created queries. The LLMs consistently outperform standard baselines and other approaches in the literature. The best performing LLMs exhibit fast response times, ranging from 1.7 to 8 seconds, and generate a manageable number of query terms (15-63 on average), making them suitable for practical implementation. Our overall findings suggest that leveraging small, open-source LLMs for clinical trials retrieval can balance performance, computational efficiency, and real-world applicability in medical settings.