How Well Do Multimodal Models Reason on ECG Signals?

TL;DR

This paper presents a scalable, reproducible framework for evaluating reasoning in multimodal models on ECG signals by separately assessing perception and deduction components, addressing limitations of prior evaluation methods.

Contribution

It introduces a novel dual-verification framework that decomposes ECG reasoning into perception and deduction, enabling more accurate assessment of true reasoning capabilities.

Findings

Effective separation of perception and deduction in evaluation.

Scalable verification of reasoning traces.

Alignment with clinical criteria demonstrated.

Abstract

While multimodal large language models offer a promising solution to the "black box" nature of health AI by generating interpretable reasoning traces, verifying the validity of these traces remains a critical challenge. Existing evaluation methods are either unscalable, relying on manual clinician review, or superficial, utilizing proxy metrics (e.g. QA) that fail to capture the semantic correctness of clinical logic. In this work, we introduce a reproducible framework for evaluating reasoning in ECG signals. We propose decomposing reasoning into two distinct, components: (i) Perception, the accurate identification of patterns within the raw signal, and (ii) Deduction, the logical application of domain knowledge to those patterns. To evaluate Perception, we employ an agentic framework that generates code to empirically verify the temporal structures described in the reasoning trace. To evaluate Deduction, we measure the alignment of the model's logic against a structured database of established clinical criteria in a retrieval-based approach. This dual-verification method enables the scalable assessment of "true" reasoning capabilities.