Contrastive Cross-Modal Learning for Infusing Chest X-ray Knowledge into ECGs

TL;DR

This paper introduces CroMoTEX, a contrastive learning framework that uses chest X-ray data during training to improve ECG-based diagnosis of cardiac conditions, enabling effective ECG analysis without needing X-ray data at test time.

Contribution

The paper presents a novel supervised cross-modal contrastive learning method that aligns ECG and CXR representations, improving ECG pathology detection without requiring X-ray data during inference.

Findings

CroMoTEX achieves up to 78.31 AUROC on edema detection.

Outperforms baseline models across multiple cardiac pathologies.

Enables scalable ECG diagnosis without X-ray data at test time.

Abstract

Modern diagnostic workflows are increasingly multimodal, integrating diverse data sources such as medical images, structured records, and physiological time series. Among these, electrocardiograms (ECGs) and chest X-rays (CXRs) are two of the most widely used modalities for cardiac assessment. While CXRs provide rich diagnostic information, ECGs are more accessible and can support scalable early warning systems. In this work, we propose CroMoTEX, a novel contrastive learning-based framework that leverages chest X-rays during training to learn clinically informative ECG representations for multiple cardiac-related pathologies: cardiomegaly, pleural effusion, and edema. Our method aligns ECG and CXR representations using a novel supervised cross-modal contrastive objective with adaptive hard negative weighting, enabling robust and task-relevant feature learning. At test time, CroMoTEX relies solely on ECG input, allowing scalable deployment in real-world settings where CXRs may be unavailable. Evaluated on the large-scale MIMIC-IV-ECG and MIMIC-CXR datasets, CroMoTEX outperforms baselines across all three pathologies, achieving up to 78.31 AUROC on edema. Our code is available at github.com/vineetpmoorty/cromotex.