Cross-Temporal Attention Fusion (CTAF) for Multimodal Physiological Signals in Self-Supervised Learning

TL;DR

This paper introduces Cross-Temporal Attention Fusion (CTAF), a novel self-supervised method for aligning and fusing asynchronous multimodal physiological signals like EEG and peripheral data, improving affect modeling accuracy with minimal labels.

Contribution

It presents a time-aware fusion mechanism with an alignment-driven self-supervised objective specifically designed for asynchronous EEG and physiological signals, along with an evaluation protocol for alignment quality.

Findings

Higher cosine margins for matched pairs

Better cross-modal token retrieval within one second

Competitive accuracy and macro-F1 with few labels

Abstract

We study multimodal affect modeling when EEG and peripheral physiology are asynchronous, which most fusion methods ignore or handle with costly warping. We propose Cross-Temporal Attention Fusion (CTAF), a self-supervised module that learns soft bidirectional alignments between modalities and builds a robust clip embedding using time-aware cross attention, a lightweight fusion gate, and alignment-regularized contrastive objectives with optional weak supervision. On the K-EmoCon dataset, under leave-one-out cross-validation evaluation, CTAF yields higher cosine margins for matched pairs and better cross-modal token retrieval within one second, and it is competitive with the baseline on three-bin accuracy and macro-F1 while using few labels. Our contributions are a time-aware fusion mechanism that directly models correspondence, an alignment-driven self-supervised objective tailored to EEG and physiology, and an evaluation protocol that measures alignment quality itself. Our approach accounts for the coupling between the central and autonomic nervous systems in psychophysiological time series. These results indicate that CTAF is a strong step toward label-efficient, generalizable EEG-peripheral fusion under temporal asynchrony.