CNNATT: Deep EEG & fNIRS Real-Time Decoding of bimanual forces

TL;DR

This paper introduces a deep learning approach combining EEG and fNIRS signals to accurately decode continuous bimanual hand forces non-invasively, advancing brain-computer interface capabilities for rehabilitation and consumer use.

Contribution

It presents the first successful non-invasive decoding of continuous bimanual forces using combined EEG and fNIRS signals with deep neural networks.

Findings

Multi-modal deep learning outperforms linear models in force decoding.

Achieved 55.2% FVAF in force reconstruction.

Improved decoding performance by at least 15% over single modalities.

Abstract

Non-invasive cortical neural interfaces have only achieved modest performance in cortical decoding of limb movements and their forces, compared to invasive brain-computer interfaces (BCIs). While non-invasive methodologies are safer, cheaper and vastly more accessible technologies, signals suffer from either poor resolution in the space domain (EEG) or the temporal domain (BOLD signal of functional Near Infrared Spectroscopy, fNIRS). The non-invasive BCI decoding of bimanual force generation and the continuous force signal has not been realised before and so we introduce an isometric grip force tracking task to evaluate the decoding. We find that combining EEG and fNIRS using deep neural networks works better than linear models to decode continuous grip force modulations produced by the left and the right hand. Our multi-modal deep learning decoder achieves 55.2 FVAF[%] in force reconstruction and improves the decoding performance by at least 15% over each individual modality. Our results show a way to achieve continuous hand force decoding using cortical signals obtained with non-invasive mobile brain imaging has immediate impact for rehabilitation, restoration and consumer applications.