Brain decoding: toward real-time reconstruction of visual perception
Yohann Benchetrit, Hubert Banville, Jean-R\'emi King
TL;DR
This paper introduces a high-temporal-resolution MEG-based decoding model that significantly improves real-time reconstruction of visual perception from brain activity, surpassing traditional fMRI methods in speed and detail.
Contribution
The study develops a novel MEG decoding framework with contrastive and regression training, enabling real-time visual perception decoding with high fidelity.
Findings
MEG decoder outperforms linear decoders by 7X in image retrieval.
Late brain responses are best decoded with DINOv2.
High-level visual features can be decoded from MEG signals.
Abstract
In the past five years, the use of generative and foundational AI systems has greatly improved the decoding of brain activity. Visual perception, in particular, can now be decoded from functional Magnetic Resonance Imaging (fMRI) with remarkable fidelity. This neuroimaging technique, however, suffers from a limited temporal resolution (0.5 Hz) and thus fundamentally constrains its real-time usage. Here, we propose an alternative approach based on magnetoencephalography (MEG), a neuroimaging device capable of measuring brain activity with high temporal resolution (5,000 Hz). For this, we develop an MEG decoding model trained with both contrastive and regression objectives and consisting of three modules: i) pretrained embeddings obtained from the image, ii) an MEG module trained end-to-end and iii) a pretrained image generator. Our results are threefold: Firstly, our…
Peer Reviews
Decision·ICLR 2024 poster
Compared to the very rich literature on methods based on MEG and other modalities, this study has increased focus on temporal resolution of the retrieval and furthermore using sota diffusion models for conditional generation. It is concluded that retrieval interesting peaks following image onset and image offset (the latter based on the after-image presumably). Retrieval performance is good for several image representations (VGG and DINOv2) The generative performance is evaluated in a number of
There is a rich literature on decoding and reconstructing visual and audio stimulus from brain recordings, so novelty is somewhat limited. Based on MEG we have high time resolution and SNR. In the temporally resolved analysis, it is interesting that VGG outperforms the more advanced representations for the direct image (after image onset) while the more complex image representations dominate retrieval based on the after-image (following image offset). We miss a discussion of this interesting fi
1. 7x improvement in decoding accuracy over linear decoders. This is an important result which will encourage neuroscience researchers to use DNNs for decoding MEG/fMRI signals. 2. Clear presentation of methods (Figure 1, Section2).
1. The reconstruction results are not impressive. Even the best examples shown in Figure 5 often do not have the reconstructions of image of same or related category. Therefore, the title is misleading as the main contribution of this paper in my opinion is DNN based MEG decoder and retrieval results and is not correctly reflected in the title. 2. The decoder is trained using a combination of two loss functions : MSE loss and CLIP loss (equation 3, line 91). There seems to be no ablation study
The work is significant in that there is no MEG decoding study that learns end-to-end to reliably generate an open set of images. Thus, it can potentially be considered a ground-breaking in this area of research, even though the techniques used are not necessarily novel from an ML perspective.
The decoding work is supposed to provide new insights to the cascade of visual processing and the unfolding of visual perception in the brain. The authors need to articulate better what insights the current observations (mentioned in the Summary) actually provide us.
Videos
No videos yet. Explain this paper in a talk, walkthrough, or lecture? Add one.
Taxonomy
TopicsFunctional Brain Connectivity Studies · CCD and CMOS Imaging Sensors · Neural dynamics and brain function