Building Safe and Deployable Clinical Natural Language Processing under Temporal Leakage Constraints

TL;DR

This paper presents a method to improve the safety and reliability of clinical NLP models by auditing and reducing temporal leakage, ensuring more trustworthy predictions in hospital discharge scenarios.

Contribution

It introduces a lightweight auditing pipeline that enhances model interpretability and reduces leakage, promoting safer deployment of clinical NLP systems.

Findings

Audited models show improved calibration and conservative predictions.

Reduction in reliance on discharge-related lexical cues.

Enhanced model robustness and safety in clinical settings.

Abstract

Clinical natural language processing (NLP) models have shown promise for supporting hospital discharge planning by leveraging narrative clinical documentation. However, note-based models are particularly vulnerable to temporal and lexical leakage, where documentation artifacts encode future clinical decisions and inflate apparent predictive performance. Such behavior poses substantial risks for real-world deployment, where overconfident or temporally invalid predictions can disrupt clinical workflows and compromise patient safety. This study focuses on system-level design choices required to build safe and deployable clinical NLP under temporal leakage constraints. We present a lightweight auditing pipeline that integrates interpretability into the model development process to identify and suppress leakage-prone signals prior to final training. Using next-day discharge prediction after elective spine surgery as a case study, we evaluate how auditing affects predictive behavior, calibration, and safety-relevant trade-offs. Results show that audited models exhibit more conservative and better-calibrated probability estimates, with reduced reliance on discharge-related lexical cues. These findings emphasize that deployment-ready clinical NLP systems should prioritize temporal validity, calibration, and behavioral robustness over optimistic performance.