A Language-Signal-Vision Multimodal Framework for Multitask Cardiac Analysis

TL;DR

This paper introduces TGMM, a comprehensive multimodal framework integrating laboratory tests, ECGs, and echocardiograms, to improve multitask cardiac analysis through dynamic data fusion and textual guidance, outperforming existing methods.

Contribution

The study presents a novel unified multimodal framework with a MedFlexFusion module and textual guidance, addressing data scarcity and modality integration challenges in cardiac analysis.

Findings

TGMM outperforms state-of-the-art methods across multiple tasks.

The framework demonstrates robustness on a public dataset.

Key modality features synergistically enhance clinical decision-making.

Abstract

Contemporary cardiovascular management involves complex consideration and integration of multimodal cardiac datasets, where each modality provides distinct but complementary physiological characteristics. While the effective integration of multiple modalities could yield a holistic clinical profile that accurately models the true clinical situation with respect to data modalities and their relatives weightings, current methodologies remain limited by: 1) the scarcity of patient- and time-aligned multimodal data; 2) reliance on isolated single-modality or rigid multimodal input combinations; 3) alignment strategies that prioritize cross-modal similarity over complementarity; and 4) a narrow single-task focus. In response to these limitations, a comprehensive multimodal dataset was curated for immediate application, integrating laboratory test results, electrocardiograms, and echocardiograms with clinical outcomes. Subsequently, a unified framework, Textual Guidance Multimodal fusion for Multiple cardiac tasks (TGMM), was proposed. TGMM incorporated three key components: 1) a MedFlexFusion module designed to capture the unique and complementary characteristics of medical modalities and dynamically integrate data from diverse cardiac sources and their combinations; 2) a textual guidance module to derive task-relevant representations tailored to diverse clinical objectives, including heart disease diagnosis, risk stratification and information retrieval; and 3) a response module to produce final decisions for all these tasks. Furthermore, this study systematically explored key features across multiple modalities and elucidated their synergistic contributions in clinical decision-making. Extensive experiments showed that TGMM outperformed state-of-the-art methods across multiple clinical tasks, with additional validation confirming its robustness on another public dataset.