HeartcareGPT: A Unified Multimodal ECG Suite for Dual Signal-Image Modeling and Understanding

TL;DR

HeartcareGPT introduces a unified multimodal ECG model with a new dataset, benchmark, and dual encoder alignment, advancing cross-modal ECG understanding and modeling.

Contribution

The paper presents HeartcareGPT, a novel multimodal ECG model with a dedicated dataset, benchmark, and a dual encoder alignment paradigm for improved ECG analysis.

Findings

HeartcareGPT achieves consistent improvements across ECG understanding tasks.

The dataset Heartcare-400K enhances model training with high-quality clinical reports.

The dual encoder projection alignment improves joint signal-image modeling.

Abstract

Although electrocardiograms (ECG) play a dominant role in cardiovascular diagnosis and treatment, their intrinsic data forms and representational patterns pose significant challenges for medical multimodal large language models (Med-MLLMs) in achieving cross-modal semantic alignment. To address this gap, we propose Heartcare Suite, a unified ECG suite designed for dual signal-image modeling and understanding: (i) Heartcare-400K. A fine-grained ECG instruction dataset on top of our data pipeline engine--HeartAgent--by integrating high quality clinical ECG reports from top hospitals with open-source data. (ii) Heartcare-Bench. A systematic benchmark assessing performance of models in multi-perspective ECG understanding and cross-modal generalization, providing guidance for optimizing ECG comprehension models. (iii) HeartcareGPT. Built upon a structure-aware discrete tokenizer Beat, we propose Dual Stream Projection Alignment (DSPA) paradigm--a dual encoder projection alignment mechanism enabling joint optimizing and modeling native ECG signal-image within a shared feature space. HeartcareGPT achieves consistent improvements across diverse ECG understanding tasks, validating both the effectiveness of the unified modeling paradigm and the necessity of a high-quality data pipeline, and establishing a methodological foundation for extending Med-MLLMs towards physiological signal domains. Our project is available at https://github.com/ZJU4HealthCare/HeartcareGPT .