ELMTEX: Fine-Tuning Large Language Models for Structured Clinical Information Extraction. A Case Study on Clinical Reports

TL;DR

This study demonstrates that fine-tuning smaller large language models can effectively extract structured clinical information from unstructured reports, improving efficiency and performance in healthcare data processing.

Contribution

The paper introduces a workflow and dataset for fine-tuning LLMs on clinical reports, showing smaller models can outperform larger ones in this domain.

Findings

Fine-tuned smaller models match or surpass larger models in accuracy.

A new dataset of 60,000 clinical summaries and translations was created.

Evaluation metrics confirmed the effectiveness of the approach.

Abstract

Europe's healthcare systems require enhanced interoperability and digitalization, driving a demand for innovative solutions to process legacy clinical data. This paper presents the results of our project, which aims to leverage Large Language Models (LLMs) to extract structured information from unstructured clinical reports, focusing on patient history, diagnoses, treatments, and other predefined categories. We developed a workflow with a user interface and evaluated LLMs of varying sizes through prompting strategies and fine-tuning. Our results show that fine-tuned smaller models match or surpass larger counterparts in performance, offering efficiency for resource-limited settings. A new dataset of 60,000 annotated English clinical summaries and 24,000 German translations was validated with automated and manual checks. The evaluations used ROUGE, BERTScore, and entity-level metrics. The work highlights the approach's viability and outlines future improvements.