A Pre-trained Framework for Multilingual Brain Decoding Using Non-invasive Recordings

TL;DR

This paper introduces a novel multilingual, multi-subject, and multimodal brain decoding framework that leverages a pre-trained multilingual model to improve speech decoding from non-invasive brain recordings across diverse languages and subjects.

Contribution

It presents a unified decoding framework that generalizes across languages, subjects, and modalities, enhancing BCI applications and promoting linguistic fairness for underrepresented languages.

Findings

Strong generalization across multilingual, multi-subject, and multimodal settings.

Boosts decoding performance for underrepresented languages.

Establishes a new paradigm for brain decoding using pre-trained models.

Abstract

Brain-computer interfaces (BCIs) with speech decoding from brain recordings have broad application potential in fields such as clinical rehabilitation and cognitive neuroscience. However, current decoding methods remain limited to single-language, single-subject, and single neuroimaging modality settings, restricting their clinical applicability and generalizability. Here we propose a joint multilingual, multi-subject and multimodal decoding framework. It maps diverse brain recordings into a unified semantic space defined by a pre-trained multilingual model (PMM), enabling decoding across multiple languages, multiple subjects and multiple neuroimaging modalities. The proposed framework is validated using non-invasive brain recordings from 159 participants across four languages. Experimental results show that it exhibits strong generalization across multilingual, multi-subject, and multimodal settings. More importantly, the proposed framework can promote linguistic fairness, which is vital for underrepresented languages in BCI applications. The unified semantic space enables cross-lingual mapping enhancement, allowing the framework to boost the decoding performance of underrepresented languages, thereby promoting linguistic fairness. Overall, the proposed framework establishes a new potential paradigm for brain decoding, opening new paths for broader applications of BCI.