Duala: Dual-Level Alignment of Subjects and Stimuli for Cross-Subject fMRI Decoding

TL;DR

Duala is a novel dual-level alignment framework that enhances cross-subject fMRI visual decoding by preserving semantic boundaries and capturing individual neural variations, significantly improving accuracy with limited data.

Contribution

It introduces a dual-level alignment approach combining semantic and subject-specific strategies, advancing the robustness and scalability of cross-subject brain decoding methods.

Findings

Achieves over 81.1% image-to-brain retrieval accuracy with minimal fine-tuning data.

Outperforms existing fine-tuning strategies in both retrieval and reconstruction tasks.

Effectively maintains semantic boundaries and captures individual neural variations.

Abstract

Cross-subject visual decoding aims to reconstruct visual experiences from brain activity across individuals, enabling more scalable and practical brain-computer interfaces. However, existing methods often suffer from degraded performance when adapting to new subjects with limited data, as they struggle to preserve both the semantic consistency of stimuli and the alignment of brain responses. To address these challenges, we propose Duala, a dual-level alignment framework designed to achieve stimulus-level consistency and subject-level alignment in fMRI-based cross-subject visual decoding. (1) At the stimulus level, Duala introduces a semantic alignment and relational consistency strategy that preserves intra-class similarity and inter-class separability, maintaining clear semantic boundaries during adaptation. (2) At the subject level, a distribution-based feature perturbation mechanism is developed to capture both global and subject-specific variations, enabling adaptation to individual neural representations without overfitting. Experiments on the Natural Scenes Dataset (NSD) demonstrate that Duala effectively improves alignment across subjects. Remarkably, even when fine-tuned with only about one hour of fMRI data, Duala achieves over 81.1% image-to-brain retrieval accuracy and consistently outperforms existing fine-tuning strategies in both retrieval and reconstruction. Our code is available at https://github.com/ShumengLI/Duala.