MindCross: Fast New Subject Adaptation with Limited Data for Cross-subject Video Reconstruction from Brain Signals

TL;DR

MindCross is a novel framework for rapid, data-efficient cross-subject video reconstruction from brain signals, effectively combining subject-specific and invariant features with a collaborative module.

Contribution

It introduces a new architecture with multiple encoders and a collaboration module for fast adaptation to new subjects using limited data.

Findings

Effective cross-subject decoding demonstrated on fMRI/EEG benchmarks.

Achieves fast adaptation with only one model and limited data.

Outperforms existing methods in efficiency and accuracy.

Abstract

Reconstructing video from brain signals is an important brain decoding task. Existing brain decoding frameworks are primarily built on a subject-dependent paradigm, which requires large amounts of brain data for each subject. However, the expensive cost of collecting brain-video data causes severe data scarcity. Although some cross-subject methods being introduced, they often overfocus with subject-invariant information while neglecting subject-specific information, resulting in slow fine-tune-based adaptation strategy. To achieve fast and data-efficient new subject adaptation, we propose MindCross, a novel cross-subject framework. MindCross's N specific encoders and one shared encoder are designed to extract subject-specific and subject-invariant information, respectively. Additionally, a Top-K collaboration module is adopted to enhance new subject decoding with the knowledge learned from previous subjects' encoders. Extensive experiments on fMRI/EEG-to-video benchmarks demonstrate MindCross's efficacy and efficiency of cross-subject decoding and new subject adaptation using only one model.