MedSlice: fine-tuned large language models for secure clinical note sectioning

TL;DR

This paper introduces MedSlice, a pipeline using open-source large language models to automatically extract clinical note sections, offering a secure and cost-effective alternative to proprietary models.

Contribution

The novel contribution is a fine-tuned open-source LLM pipeline for clinical note sectioning that outperforms proprietary models in performance and privacy.

Findings

Fine-tuned Llama 3.1 8B achieved an F1 score of 0.92, outperforming GPT-4o in clinical note sectioning.

The model maintained high performance (F1 = 0.85) on an external test set, demonstrating robust generalizability.

Open-source models provide privacy advantages and cost-effectiveness compared to proprietary alternatives in clinical settings.

Abstract

Extracting sections from clinical notes is crucial for downstream analysis but is challenging due to variability in formatting and labor-intensive nature of manual sectioning. This study develops a pipeline for automated note sectioning using open-source large language models (LLMs), focusing on three sections: History of Present Illness, Interval History, and Assessment and Plan.

We fine-tuned three open-source LLMs to extract sections using a curated dataset of 487 progress notes, comparing results relative to proprietary models (GPT-4o, GPT-4o mini). Internal and external validity were assessed via precision, recall, and F1 score.

Fine-tuned Llama 3.1 8B (F1 = 0.92) outperformed GPT-4o. On the external validity test set, performance remained high (F1 = 0.85).

While proprietary LLMs have shown promise, privacy concerns limit their utility in medicine; fine-tuned, open-source LLMs offer advantages in cost, performance, and accessibility.

Fine-tuned, open-source LLMs can surpass proprietary models in clinical note sectioning.

Figures