Large Language Model-Based Generation of Discharge Summaries

TL;DR

This study evaluates various large language models for automatically generating discharge summaries from medical notes, finding proprietary models like Gemini excel in accuracy and utility, though challenges like hallucinations remain.

Contribution

It compares open-source and proprietary LLMs on discharge summary generation, highlighting the superior performance of proprietary models with one-shot prompting.

Findings

Proprietary models outperform open-source models in summary accuracy.

Fine-tuning improves open-source model performance.

Human evaluation confirms the clinical utility of proprietary model summaries.

Abstract

Discharge Summaries are documents written by medical professionals that detail a patient's visit to a care facility. They contain a wealth of information crucial for patient care, and automating their generation could significantly reduce the effort required from healthcare professionals, minimize errors, and ensure that critical patient information is easily accessible and actionable. In this work, we explore the use of five Large Language Models on this task, from open-source models (Mistral, Llama 2) to proprietary systems (GPT-3, GPT-4, Gemini 1.5 Pro), leveraging MIMIC-III summaries and notes. We evaluate them using exact-match, soft-overlap, and reference-free metrics. Our results show that proprietary models, particularly Gemini with one-shot prompting, outperformed others, producing summaries with the highest similarity to the gold-standard ones. Open-source models, while promising, especially Mistral after fine-tuning, lagged in performance, often struggling with hallucinations and repeated information. Human evaluation by a clinical expert confirmed the practical utility of the summaries generated by proprietary models. Despite the challenges, such as hallucinations and missing information, the findings suggest that LLMs, especially proprietary models, are promising candidates for automatic discharge summary generation as long as data privacy is ensured.