Automated Cardiovascular Record Retrieval by Multimodal Learning between Electrocardiogram and Clinical Report

TL;DR

This paper presents a novel multimodal learning approach combining ECG images and clinical reports using vision-language models to improve automated ECG interpretation and retrieval, especially benefiting underdeveloped regions.

Contribution

It introduces a new ECG interpretation method that leverages vision-language models for case retrieval, moving beyond traditional classification tasks.

Findings

Effective ECG retrieval system demonstrated

Improved diagnostic support for underdeveloped regions

Joint learning of ECG images and reports enhances accuracy

Abstract

Automated interpretation of electrocardiograms (ECG) has garnered significant attention with the advancements in machine learning methodologies. Despite the growing interest, most current studies focus solely on classification or regression tasks, which overlook a crucial aspect of clinical cardio-disease diagnosis: the diagnostic report generated by experienced human clinicians. In this paper, we introduce a novel approach to ECG interpretation, leveraging recent breakthroughs in Large Language Models (LLMs) and Vision-Transformer (ViT) models. Rather than treating ECG diagnosis as a classification or regression task, we propose an alternative method of automatically identifying the most similar clinical cases based on the input ECG data. Also, since interpreting ECG as images is more affordable and accessible, we process ECG as encoded images and adopt a vision-language learning paradigm to jointly learn vision-language alignment between encoded ECG images and ECG diagnosis reports. Encoding ECG into images can result in an efficient ECG retrieval system, which will be highly practical and useful in clinical applications. More importantly, our findings could serve as a crucial resource for providing diagnostic services in underdeveloped regions.