Systematic Evaluation of Large Language Models for Post-Discharge Clinical Action Extraction

TL;DR

This paper systematically evaluates large language models for extracting post-discharge clinical actions from narrative discharge notes, comparing their performance to supervised models and analyzing annotation challenges.

Contribution

It introduces a two-stage extraction framework and provides a comprehensive assessment of LLMs versus supervised models in clinical action extraction tasks.

Findings

LLMs perform comparably to supervised models on binary action detection.

Supervised models outperform LLMs on multi-label category classification.

Annotation inconsistencies and lack of reasoning annotations hinder model evaluation.

Abstract

The work in this paper evaluates zero-shot and few-shot large language models (LLMs) for safety-critical clinical action extraction using the CLIP discharge-note dataset, with particular emphasis on transitions of care and post-discharge patient safety. To manage the complexity of clinical documentation, we introduce a two-stage extraction framework that decomposes discharge notes, that are written in narrative form, into fine-grained, explicitly actionable clinical tasks through a staged prompting strategy. Our contributions include a systematic assessment of generative LLMs for clinical action extraction, a detailed comparison between general-purpose LLMs and task-specific supervised BERT-based models, and an analysis of annotation inconsistencies across different action categories. We show that contemporary LLMs achieve performance comparable to or exceeding supervised models on binary actionability detection, while supervised baselines retain a meaningful advantage on fine-grained multi-label category classification, despite the absence of task-specific fine-tuning and under strict data-privacy constraints. Qualitative error analysis reveals that many failures stem from misalignment between model reasoning and dataset annotation conventions, particularly in cases involving implicit clinical actions and rigid structural labeling rules. These results indicate that reported performance reflects model limitations due to lack of clinical reasoning, that is not captured by plain annotations. Labels without rationales make it impossible to distinguish clinical reasoning failures from annotation convention mismatches. Advancing clinical NLP requires reasoning-annotated datasets that document why specific spans are actionable, not merely which spans were labeled, enabling proper evaluation of model clinical understanding.