Neuro2Semantic: A Transfer Learning Framework for Semantic Reconstruction of Continuous Language from Human Intracranial EEG
Siavash Shams, Richard Antonello, Gavin Mischler, Stephan Bickel, Ashesh Mehta, and Nima Mesgarani
TL;DR
Neuro2Semantic is a transfer learning framework that reconstructs continuous speech semantics from intracranial EEG, enabling natural text generation with limited neural data and outperforming previous methods.
Contribution
The paper introduces Neuro2Semantic, a novel two-phase framework that aligns neural signals with text embeddings and generates natural language, advancing neural decoding capabilities.
Findings
Achieves strong performance with only 30 minutes of neural data
Outperforms recent state-of-the-art in low-data settings
Enables unconstrained, natural text generation from neural signals
Abstract
Decoding continuous language from neural signals remains a significant challenge in the intersection of neuroscience and artificial intelligence. We introduce Neuro2Semantic, a novel framework that reconstructs the semantic content of perceived speech from intracranial EEG (iEEG) recordings. Our approach consists of two phases: first, an LSTM-based adapter aligns neural signals with pre-trained text embeddings; second, a corrector module generates continuous, natural text directly from these aligned embeddings. This flexible method overcomes the limitations of previous decoding approaches and enables unconstrained text generation. Neuro2Semantic achieves strong performance with as little as 30 minutes of neural data, outperforming a recent state-of-the-art method in low-data settings. These results highlight the potential for practical applications in brain-computer interfaces and…
Peer Reviews
Decision·ICLR 2025 Conference Withdrawn Submission
1. The transfer learning framework adopted by Neuro2Semantic can achieve efficient semantic reconstruction under the condition of a small amount of data. 2. Neuro2Semantic is able to perform zero-shot reconstruction on completely unseen semantic content, showing good generalization.
1. This paper only compares one baseline, and this baseline is very poor. 2. Model performance is highly dependent on the quality and relevance of pre-trained text embeddings, which may limit its effectiveness when dealing with domain-specific or rare vocabularies.
The performance reported in the experimental section shows good improvement over baseline models. It shows the BLEU score reached 0.1947 while the baseline method only shows 0.0315.
1. The experiment is not adequate to claim that this is a solid SOTA for brian-to-text translation, especially only given 30 minutes of training data. Considering this method has a T5 corrector over the previous method, the well-trained language model will generate multiple semantically continuous sentences containing many articles and common words will give the illusion of an artificially high bleuscore. Figure 2 generated text samples also support this phenomenon. Meanwhile, do the training se
The submission is well written and clear. Authors conducted comprehensive experiments. The results demonstrates the effectivness of the methods outperforms the baseline method the scenario for in domain and out of domain cases.
Generally, the semantic meaning of the reconstructed sentences in both Neuro2Semantic and the baseline method appears to diverge significantly from the original text, as illustrated in Figures 2B and 3B. While the word error rate (WER) captures only surface-level matching between the decoded sentences and the ground truth, it is crucial to retain certain key nouns to ensure accurate semantic transmission. Current word error rate is very high, also other metrics like BERT score or BLEU score is q
Code & Models
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsNeural Networks and Applications · Topic Modeling · EEG and Brain-Computer Interfaces
MethodsAdapter