Leveraging Foundational Models and Simple Fusion for Multi-modal Physiological Signal Analysis

TL;DR

This paper introduces a foundation-model-based approach for multi-modal physiological signal analysis, using simple fusion of pre-trained ECG and EEG encoders to improve emotion recognition with limited labeled data.

Contribution

It adapts the CBraMod encoder for large-scale self-supervised ECG pretraining and combines it with a pre-trained EEG encoder, enabling effective multi-modal integration with simple concatenation.

Findings

Achieves near state-of-the-art emotion recognition performance.

Demonstrates the effectiveness of foundation models in physiological signal analysis.

Shows simple embedding fusion can leverage multi-modal representations effectively.

Abstract

Physiological signals such as electrocardiograms (ECG) and electroencephalograms (EEG) provide complementary insights into human health and cognition, yet multi-modal integration is challenging due to limited multi-modal labeled data, and modality-specific differences . In this work, we adapt the CBraMod encoder for large-scale self-supervised ECG pretraining, introducing a dual-masking strategy to capture intra- and inter-lead dependencies. To overcome the above challenges, we utilize a pre-trained CBraMod encoder for EEG and pre-train a symmetric ECG encoder, equipping each modality with a rich foundational representation. These representations are then fused via simple embedding concatenation, allowing the classification head to learn cross-modal interactions, together enabling effective downstream learning despite limited multi-modal supervision. Evaluated on emotion recognition, our approach achieves near state-of-the-art performance, demonstrating that carefully designed physiological encoders, even with straightforward fusion, substantially improve downstream performance. These results highlight the potential of foundation-model approaches to harness the holistic nature of physiological signals, enabling scalable, label-efficient, and generalizable solutions for healthcare and affective computing.