Cross-Subject Mind Decoding from Inaccurate Representations

TL;DR

This paper introduces a bidirectional autoencoder framework that improves cross-subject fMRI-based stimulus image decoding by addressing inaccuracies and variability, achieving superior results and adaptability.

Contribution

It proposes a novel bidirectional autoencoder with modules for bias modulation, semantic refinement, and visual coherence, enhancing decoding accuracy and generalization across subjects.

Findings

Outperforms state-of-the-art methods on benchmark datasets.

Achieves high fidelity in stimulus image reconstruction.

Demonstrates strong adaptability to new subjects with minimal data.

Abstract

Decoding stimulus images from fMRI signals has advanced with pre-trained generative models. However, existing methods struggle with cross-subject mappings due to cognitive variability and subject-specific differences. This challenge arises from sequential errors, where unidirectional mappings generate partially inaccurate representations that, when fed into diffusion models, accumulate errors and degrade reconstruction fidelity. To address this, we propose the Bidirectional Autoencoder Intertwining framework for accurate decoded representation prediction. Our approach unifies multiple subjects through a Subject Bias Modulation Module while leveraging bidirectional mapping to better capture data distributions for precise representation prediction. To further enhance fidelity when decoding representations into stimulus images, we introduce a Semantic Refinement Module to improve semantic representations and a Visual Coherence Module to mitigate the effects of inaccurate visual representations. Integrated with ControlNet and Stable Diffusion, our method outperforms state-of-the-art approaches on benchmark datasets in both qualitative and quantitative evaluations. Moreover, our framework exhibits strong adaptability to new subjects with minimal training samples.