Wills Aligner: Multi-Subject Collaborative Brain Visual Decoding

TL;DR

Wills Aligner is a novel multi-subject brain decoding method that aligns fMRI data and uses meta-learning to improve visual decoding across individuals, enhancing applicability in real-world scenarios.

Contribution

It introduces a new approach combining anatomical alignment, mixture-of-brain-expert adapters, and meta-learning for collaborative brain decoding across subjects.

Findings

Achieves promising performance in classification tasks

Effective in cross-modal retrieval scenarios

Improves image reconstruction accuracy

Abstract

Decoding visual information from human brain activity has seen remarkable advancements in recent research. However, the diversity in cortical parcellation and fMRI patterns across individuals has prompted the development of deep learning models tailored to each subject. The personalization limits the broader applicability of brain visual decoding in real-world scenarios. To address this issue, we introduce Wills Aligner, a novel approach designed to achieve multi-subject collaborative brain visual decoding. Wills Aligner begins by aligning the fMRI data from different subjects at the anatomical level. It then employs delicate mixture-of-brain-expert adapters and a meta-learning strategy to account for individual fMRI pattern differences. Additionally, Wills Aligner leverages the semantic relation of visual stimuli to guide the learning of inter-subject commonality, enabling visual decoding for each subject to draw insights from other subjects' data. We rigorously evaluate our Wills Aligner across various visual decoding tasks, including classification, cross-modal retrieval, and image reconstruction. The experimental results demonstrate that Wills Aligner achieves promising performance.