HeartLLM: Discretized ECG Tokenization for LLM-Based Diagnostic Reasoning

TL;DR

HeartLLM introduces a novel method to enable large language models to process ECG signals by discretizing continuous ECG data into tokens, improving diagnostic reasoning and generalization across clinical tasks.

Contribution

The paper presents a new framework that integrates ECG signal processing with LLMs through discretization and tokenization, facilitating open-ended medical reasoning without modifying core models.

Findings

Achieves strong performance on ECG question answering and report generation

Maintains generalization to out-of-distribution data

Demonstrates effectiveness of discretized ECG tokens in medical reasoning

Abstract

Electrocardiography (ECG) plays a central role in cardiovascular diagnostics, yet existing automated approaches often struggle to generalize across clinical tasks and offer limited support for open-ended reasoning. We present HeartLLM, a novel framework that integrates time-series (TS) and language modeling by enabling large language models (LLMs) to process 12-lead ECG signals for clinical text generation tasks. Our approach discretizes continuous ECG embeddings into quantized codes using a lead-wise encoder and quantization module. These quantized codes are then mapped to an extended ECG vocabulary to form ECG tokens, enabling the model to process both ECG and natural language inputs within a unified framework. To bridge the modality gap, we pretrain the model on an autoregressive ECG token forecasting task, allowing the LLM to capture temporal dynamics through its inherent language modeling capability. Finally, we perform instruction tuning on both ECG question answering and diagnostic report generation. Without modifying the core model, HeartLLM achieves strong performance across tasks while maintaining generalization to out-of-distribution settings. Extensive experiments demonstrate the effectiveness of each component and highlight the potential of integrating discretized ECG tokens into LLMs for medical reasoning.