Leveraging Open-Source Large Language Models for Clinical Information Extraction in Resource-Constrained Settings

TL;DR

This study evaluates open-source large language models for extracting clinical information from Dutch medical reports, demonstrating their effectiveness and scalability in resource-limited healthcare settings.

Contribution

It introduces exttt{llm extunderscore extractinator}, a framework for clinical info extraction with open-source LLMs, and benchmarks their performance on the Dutch DRAGON dataset.

Findings

Several 14B parameter models achieved competitive results.

Larger models like Llama-3.3-70B perform slightly better but require more resources.

Translation to English reduces model performance, emphasizing the importance of native language processing.

Abstract

Medical reports contain rich clinical information but are often unstructured and written in domain-specific language, posing challenges for information extraction. While proprietary large language models (LLMs) have shown promise in clinical natural language processing, their lack of transparency and data privacy concerns limit their utility in healthcare. This study therefore evaluates nine open-source generative LLMs on the DRAGON benchmark, which includes 28 clinical information extraction tasks in Dutch. We developed \texttt{llm\_extractinator}, a publicly available framework for information extraction using open-source generative LLMs, and used it to assess model performance in a zero-shot setting. Several 14 billion parameter models, Phi-4-14B, Qwen-2.5-14B, and DeepSeek-R1-14B, achieved competitive results, while the bigger Llama-3.3-70B model achieved slightly higher performance at greater computational cost. Translation to English prior to inference consistently degraded performance, highlighting the need of native-language processing. These findings demonstrate that open-source LLMs, when used with our framework, offer effective, scalable, and privacy-conscious solutions for clinical information extraction in low-resource settings.