UniCoRN: Unified Cognitive Signal ReconstructioN bridging cognitive signals and human language

TL;DR

This paper introduces fMRI2text, an open-vocabulary task for decoding human language from cognitive signals like fMRI and EEG, and presents UniCoRN, a baseline model that effectively reconstructs coherent text, demonstrating the potential for versatile brain-computer interfaces.

Contribution

It proposes the first open-vocabulary fMRI2text task and introduces UniCoRN, a unified model capable of decoding cognitive signals into human language across multiple modalities.

Findings

Achieved 34.77% BLEU on fMRI2text

Achieved 37.04% BLEU on EEG-to-text decoding

Demonstrated the feasibility of decoding consecutive fMRI volumes

Abstract

Decoding text stimuli from cognitive signals (e.g. fMRI) enhances our understanding of the human language system, paving the way for building versatile Brain-Computer Interface. However, existing studies largely focus on decoding individual word-level fMRI volumes from a restricted vocabulary, which is far too idealized for real-world application. In this paper, we propose fMRI2text, the first openvocabulary task aiming to bridge fMRI time series and human language. Furthermore, to explore the potential of this new task, we present a baseline solution, UniCoRN: the Unified Cognitive Signal ReconstructioN for Brain Decoding. By reconstructing both individual time points and time series, UniCoRN establishes a robust encoder for cognitive signals (fMRI & EEG). Leveraging a pre-trained language model as decoder, UniCoRN proves its efficacy in decoding coherent text from fMRI series across various split settings. Our model achieves a 34.77% BLEU score on fMRI2text, and a 37.04% BLEU when generalized to EEGto-text decoding, thereby surpassing the former baseline. Experimental results indicate the feasibility of decoding consecutive fMRI volumes, and the effectiveness of decoding different cognitive signals using a unified structure.