UniPACT: A Multimodal Framework for Prognostic Question Answering on Raw ECG and Structured EHR

TL;DR

UniPACT is a multimodal framework that combines structured EHR data and raw ECG signals to improve prognostic question answering in clinical settings, achieving state-of-the-art results.

Contribution

It introduces a structured prompting mechanism that textualizes EHR data and fuses it with ECG representations for holistic reasoning by LLMs.

Findings

Achieves a mean AUROC of 89.37% on MDS-ED benchmark.

Outperforms specialized baselines across multiple prognostic tasks.

Demonstrates robustness in scenarios with missing data.

Abstract

Accurate clinical prognosis requires synthesizing structured Electronic Health Records (EHRs) with real-time physiological signals like the Electrocardiogram (ECG). Large Language Models (LLMs) offer a powerful reasoning engine for this task but struggle to natively process these heterogeneous, non-textual data types. To address this, we propose UniPACT (Unified Prognostic Question Answering for Clinical Time-series), a unified framework for prognostic question answering that bridges this modality gap. UniPACT's core contribution is a structured prompting mechanism that converts numerical EHR data into semantically rich text. This textualized patient context is then fused with representations learned directly from raw ECG waveforms, enabling an LLM to reason over both modalities holistically. We evaluate UniPACT on the comprehensive MDS-ED benchmark, it achieves a state-of-the-art mean AUROC of 89.37% across a diverse set of prognostic tasks including diagnosis, deterioration, ICU admission, and mortality, outperforming specialized baselines. Further analysis demonstrates that our multimodal, multi-task approach is critical for performance and provides robustness in missing data scenarios.