Identifying Symptoms of Delirium from Clinical Narratives Using Natural Language Processing

TL;DR

This study develops and evaluates NLP methods, especially transformer models, to automatically identify delirium symptoms from clinical narratives, aiding diagnosis and phenotyping in electronic health records.

Contribution

It introduces the first large language model-based system for delirium symptom extraction from clinical notes, utilizing a newly created delirium corpus and expert annotations.

Findings

GatorTron achieved the highest F1 scores of 0.8055 (strict) and 0.8759 (lenient).

Transformer models outperform traditional NLP approaches in delirium symptom extraction.

Error analysis highlights challenges in annotation and model development.

Abstract

Delirium is an acute decline or fluctuation in attention, awareness, or other cognitive function that can lead to serious adverse outcomes. Despite the severe outcomes, delirium is frequently unrecognized and uncoded in patients' electronic health records (EHRs) due to its transient and diverse nature. Natural language processing (NLP), a key technology that extracts medical concepts from clinical narratives, has shown great potential in studies of delirium outcomes and symptoms. To assist in the diagnosis and phenotyping of delirium, we formed an expert panel to categorize diverse delirium symptoms, composed annotation guidelines, created a delirium corpus with diverse delirium symptoms, and developed NLP methods to extract delirium symptoms from clinical notes. We compared 5 state-of-the-art transformer models including 2 models (BERT and RoBERTa) from the general domain and 3 models (BERT_MIMIC, RoBERTa_MIMIC, and GatorTron) from the clinical domain. GatorTron achieved the best strict and lenient F1 scores of 0.8055 and 0.8759, respectively. We conducted an error analysis to identify challenges in annotating delirium symptoms and developing NLP systems. To the best of our knowledge, this is the first large language model-based delirium symptom extraction system. Our study lays the foundation for the future development of computable phenotypes and diagnosis methods for delirium.