Phase 1 Implementation of LLM-generated Discharge Summaries showing high Adoption in a Dutch Academic Hospital

TL;DR

This study piloted an LLM-based tool for generating discharge summaries in a Dutch hospital, showing high adoption, reduced documentation time, and strong intent for continued use.

Contribution

It demonstrates the feasibility and acceptance of integrating LLMs into clinical workflows for discharge summaries in a real-world hospital setting.

Findings

58.5% of discharge summaries included LLM-generated text

86.9% of users reported reduced documentation time

91.3% intended to continue using the system

Abstract

Writing discharge summaries to transfer medical information is an important but time-consuming process that can be assisted by Large Language Models (LLMs). This prospective mixed methods pilot study evaluated an Electronic Health Record (EHR)-integrated LLM to generate discharge summaries drafts. In total, 379 discharge summaries were generated in clinical practice by 21 residents and 4 physician assistants during 9 weeks in our academic hospital. LLM-generated text was copied in 58.5% of admissions, and identifiable LLM content could be traced to 29.1% of final discharge letters. Notably, 86.9% of users self-reported a reduction in documentation time, and 60.9% a reduction in administrative workload. Intent to use after the pilot phase was high (91.3%), supporting further implementation of this use-case. Accurately measuring the documentation time of users on discharge summaries remains challenging, but will be necessary for future extrinsic evaluation of LLM-assisted documentation.