Narrative Feature or Structured Feature? A Study of Large Language Models to Identify Cancer Patients at Risk of Heart Failure

TL;DR

This study evaluates machine learning models, including large language models, to predict heart failure risk in cancer patients using electronic health records, demonstrating that narrative features significantly enhance prediction accuracy.

Contribution

It introduces novel narrative features derived from structured medical codes and compares their effectiveness across multiple ML models, highlighting the superior performance of GatorTron-3.9B.

Findings

GatorTron-3.9B achieved the highest F1 scores among models.

Narrative features increased feature density and improved model performance.

LLMs outperformed traditional ML and other transformer models in predicting HF risk.

Abstract

Cancer treatments are known to introduce cardiotoxicity, negatively impacting outcomes and survivorship. Identifying cancer patients at risk of heart failure (HF) is critical to improving cancer treatment outcomes and safety. This study examined machine learning (ML) models to identify cancer patients at risk of HF using electronic health records (EHRs), including traditional ML, Time-Aware long short-term memory (T-LSTM), and large language models (LLMs) using novel narrative features derived from the structured medical codes. We identified a cancer cohort of 12,806 patients from the University of Florida Health, diagnosed with lung, breast, and colorectal cancers, among which 1,602 individuals developed HF after cancer. The LLM, GatorTron-3.9B, achieved the best F1 scores, outperforming the traditional support vector machines by 39%, the T-LSTM deep learning model by 7%, and a widely used transformer model, BERT, by 5.6%. The analysis shows that the proposed narrative features remarkably increased feature density and improved performance.