Large Language Model-Assisted Point-in-Time Interpretation of Advanced Hemodynamics in Liver Transplant Recipients: A Pilot Evaluation of Content Quality and Safety

TL;DR

This study explores how well ChatGPT can interpret complex hemodynamic data during liver transplants, finding it generally reliable with few safety concerns.

Contribution

The study introduces a novel evaluation framework for assessing LLMs in interpreting advanced hemodynamic monitoring during liver transplantation.

Findings

ChatGPT showed high performance in interpreting hemodynamic data across multiple quality domains.

Only a small subset of responses received lower scores, typically during abrupt physiological changes.

The ARQuAT framework demonstrated strong internal consistency but modest inter-rater agreement.

Abstract

Background: Large language models (LLMs) are increasingly used in clinical medicine, yet their ability to interpret advanced intraoperative hemodynamic monitoring—particularly in the context of liver transplantation—remains largely unexplored. In this proof-of-concept study, we evaluated ChatGPT’s capacity to interpret multimodal hemodynamic data derived from both standard anesthesia monitoring and the PiCCO system. The study also employed a structured assessment instrument (ARQuAT), adapted through a Delphi-based process to evaluate LLM-generated clinical interpretations. Methods: Ten key surgical–hemodynamic phases of liver transplantation were identified using a modified Delphi approach to capture the major physiological transitions of the procedure. Sequential screenshots representing these phases were obtained from five liver transplant recipients, yielding a total of 50 images. Each screenshot, along with standardized clinical background information, was submitted to ChatGPT. Five expert anesthesiologists independently assessed the model’s responses using the modified ARQuAT tool, which includes six content-quality domains (Accuracy, Up-to-dateness, Contextual Consistency, Clinical Usability, Trustworthiness, Clarity) and a separate catastrophic Risk item. Descriptive statistics were calculated for domain-level performance. Inter-rater reliability (Kendall’s W) and internal consistency (Cronbach’s alpha, McDonald’s omega) were also analyzed. All statistical analyses and visualizations were performed using NumIQO. Results: ChatGPT demonstrated consistently high performance across all content-quality domains, with median scores ranging from 4.6 to 4.8 and more than 90% of all ratings classified as satisfactory. Lower scores appeared only in a small subset of frames associated with abrupt hemodynamic changes and did not indicate a recurring weakness in any specific domain. Catastrophic Risk exhibited a pronounced floor effect, with 86% of ratings scored as 0 and only three isolated high-risk assessments across the dataset. Internal consistency of the six ARQuAT content domains was excellent, while inter-rater agreement was modest, reflecting ceiling effects and tied ratings among evaluators. Conclusions: ChatGPT generated clinically acceptable, contextually aligned interpretations of complex intraoperative hemodynamic data in liver transplant recipients, with minimal evidence of unsafe recommendations. These findings suggest preliminary promise for LLM-assisted interpretation of advanced monitoring, while underscoring the need for future studies involving larger datasets, dynamic physiological inputs, and expanded evaluator groups. The reliability characteristics observed also provide initial support for further refinement and broader validation of the Delphi-derived ARQuAT framework.

Figures