SPEER: Sentence-Level Planning of Long Clinical Summaries via Embedded Entity Retrieval

TL;DR

This paper introduces SPEER, a sentence-level planning method using embedded entity retrieval to improve the coverage and faithfulness of long clinical summaries generated by fine-tuned large language models.

Contribution

It proposes a novel sentence-level planning approach with embedded entity retrieval to enhance clinical summary generation, addressing issues of incompleteness and unfaithfulness.

Findings

SPEER improves entity coverage in generated summaries.

SPEER enhances faithfulness metrics over baselines.

The method is effective across multiple datasets.

Abstract

Clinician must write a lengthy summary each time a patient is discharged from the hospital. This task is time-consuming due to the sheer number of unique clinical concepts covered in the admission. Identifying and covering salient entities is vital for the summary to be clinically useful. We fine-tune open-source LLMs (Mistral-7B-Instruct and Zephyr-7B-beta) on the task and find that they generate incomplete and unfaithful summaries. To increase entity coverage, we train a smaller, encoder-only model to predict salient entities, which are treated as content-plans to guide the LLM. To encourage the LLM to focus on specific mentions in the source notes, we propose SPEER: Sentence-level Planning via Embedded Entity Retrieval. Specifically, we mark each salient entity span with special "{{ }}" boundary tags and instruct the LLM to retrieve marked spans before generating each sentence. Sentence-level planning acts as a form of state tracking in that the model is explicitly recording the entities it uses. We fine-tune Mistral and Zephyr variants on a large-scale, diverse dataset of ~167k in-patient hospital admissions and evaluate on 3 datasets. SPEER shows gains in both coverage and faithfulness metrics over non-guided and guided baselines.