e-Health CSIRO at "Discharge Me!" 2024: Generating Discharge Summary Sections with Fine-tuned Language Models

TL;DR

This paper explores fine-tuning various open-source language models to automatically generate discharge summary sections, aiming to reduce clinicians' documentation workload in the BioNLP 2024 Shared Task.

Contribution

It systematically evaluates different language model configurations and decoding strategies for discharge summary section generation, highlighting the effectiveness of content conditioning and smaller encoder-decoder models.

Findings

Conditioning on prior discharge content improves generation quality.

Smaller encoder-decoder models perform comparably or better than larger decoder-only models.

Ensembling and specialized models enhance output accuracy.

Abstract

Clinical documentation is an important aspect of clinicians' daily work and often demands a significant amount of time. The BioNLP 2024 Shared Task on Streamlining Discharge Documentation (Discharge Me!) aims to alleviate this documentation burden by automatically generating discharge summary sections, including brief hospital course and discharge instruction, which are often time-consuming to synthesize and write manually. We approach the generation task by fine-tuning multiple open-sourced language models (LMs), including both decoder-only and encoder-decoder LMs, with various configurations on input context. We also examine different setups for decoding algorithms, model ensembling or merging, and model specialization. Our results show that conditioning on the content of discharge summary prior to the target sections is effective for the generation task. Furthermore, we find that smaller encoder-decoder LMs can work as well or even slightly better than larger decoder based LMs fine-tuned through LoRA. The model checkpoints from our team (aehrc) are openly available.