NT-ViT: Neural Transcoding Vision Transformers for EEG-to-fMRI Synthesis

TL;DR

NT-ViT is a novel generative model that synthesizes high-resolution fMRI images from EEG data, improving accuracy and reproducibility over previous methods, with significant performance gains demonstrated on benchmark datasets.

Contribution

The paper introduces NT-ViT, a new neural transcoding model with a domain matching module that enhances EEG-to-fMRI synthesis accuracy and reliability, outperforming existing approaches.

Findings

Achieves 10x reduction in RMSE on the Oddball dataset.

Increases SSIM by 3.14x compared to state-of-the-art.

Demonstrates superior performance on two benchmark datasets.

Abstract

This paper introduces the Neural Transcoding Vision Transformer (\modelname), a generative model designed to estimate high-resolution functional Magnetic Resonance Imaging (fMRI) samples from simultaneous Electroencephalography (EEG) data. A key feature of \modelname is its Domain Matching (DM) sub-module which effectively aligns the latent EEG representations with those of fMRI volumes, enhancing the model's accuracy and reliability. Unlike previous methods that tend to struggle with fidelity and reproducibility of images, \modelname addresses these challenges by ensuring methodological integrity and higher-quality reconstructions which we showcase through extensive evaluation on two benchmark datasets; \modelname outperforms the current state-of-the-art by a significant margin in both cases, e.g. achieving a $10\times$ reduction in RMSE and a $3.14\times$ increase in SSIM on the Oddball dataset. An ablation study also provides insights into the contribution of each component to the model's overall effectiveness. This development is critical in offering a new approach to lessen the time and financial constraints typically linked with high-resolution brain imaging, thereby aiding in the swift and precise diagnosis of neurological disorders. Although it is not a replacement for actual fMRI but rather a step towards making such imaging more accessible, we believe that it represents a pivotal advancement in clinical practice and neuroscience research. Code is available at \url{https://github.com/rom42pla/ntvit}.