# Predicting Postpartum Hemorrhage Using Clinical Features Extracted With Large Language Models

## TL;DR

This study shows that using large language models to analyze clinical notes can help predict postpartum hemorrhage before labor begins, improving early identification of at-risk patients.

## Contribution

The study introduces a novel intervention-based definition of postpartum hemorrhage and demonstrates the effectiveness of LLM-based prediction models using clinical notes.

## Key findings

- LLM-based direct prediction models outperformed structured data models in predicting postpartum hemorrhage.
- The LLM-extracted features approach identified 47 significant predictors, including known risk factors like multiple gestation and previous cesarean delivery.
- Models using LLM-extracted features combined with structured data achieved a balance between predictive performance and clinical utility.

## Abstract

Feature extraction from clinical notes using large language models can be used to predict postpartum hemorrhage before the onset of labor, supporting early identification of at-risk patients.

To evaluate whether large language models (LLMs) applied to prenatal clinical notes can predict postpartum hemorrhage (PPH) before the onset of labor and to compare model performance across outcome definitions, including a novel intervention-based definition.

We conducted a retrospective cohort study within a large regional health network. Two outcome definitions for PPH were used: 1) estimated or quantitative blood loss (EBL–QBL) extracted from clinical notes; and 2) a clinical intervention–based PPH definition (cPPH) designed to capture significant hemorrhage requiring intervention, including transfusion, uterotonics, Bakri balloon, or hysterectomy. We evaluated three PPH prediction pipelines: 1) structured data only—supervised machine learning that used structured electronic medical record data; 2) LLM-direct—direct prediction that used a fine-tuned LLM applied to clinical notes; and 3) LLM-extract—interpretable models that used LLM-extracted features combined with structured data. Model performance was evaluated using an area under the receiver operating characteristic curve (AUROC) on a temporally held-out test set.

Among 19,992 deliveries, 1,156 patients (5.8%) met the EBL–QBL definition of PPH, 321 (1.6%) met the cPPH definition, and 309 (1.5%) met both definitions. The LLM-based direct prediction model achieved the highest AUROC for both PPH definitions (AUROC 0.79–0.80), followed by interpretable models that combined LLM-extracted features with structured data (AUROC 0.76–0.78). Models that used only structured data had the lowest AUROC (0.65–0.71). The LLM-extracted features approach identified 47 significant predictors, including established risk factors such as multiple gestation and previous cesarean delivery.

These findings highlight the potential of LLM-based approaches to improve PPH risk stratification beyond structured data alone, with the feature extraction method offering a promising balance between predictive performance and clinical utility. Eventual integration of these methods into clinical workflows could improve early detection and guide targeted preventive interventions.