Multimodal Classification with Deep Convolutional-Recurrent Neural Networks for Electroencephalography

TL;DR

This paper introduces a novel deep learning approach that models EEG data as video classification, integrating multimodal information with CNN and RNN, leading to improved accuracy and robustness in brain-computer interface applications.

Contribution

It proposes a new EEG classification method using video modeling and optical flow, combining CNN and RNN to better exploit multimodal EEG data.

Findings

Enhanced classification accuracy over traditional methods

Improved robustness in EEG classification tasks

Effective application in stroke rehabilitation support

Abstract

Electroencephalography (EEG) has become the most significant input signal for brain computer interface (BCI) based systems. However, it is very difficult to obtain satisfactory classification accuracy due to traditional methods can not fully exploit multimodal information. Herein, we propose a novel approach to modeling cognitive events from EEG data by reducing it to a video classification problem, which is designed to preserve the multimodal information of EEG. In addition, optical flow is introduced to represent the variant information of EEG. We train a deep neural network (DNN) with convolutional neural network (CNN) and recurrent neural network (RNN) for the EEG classification task by using EEG video and optical flow. The experiments demonstrate that our approach has many advantages, such as more robustness and more accuracy in EEG classification tasks. According to our approach, we designed a mixed BCI-based rehabilitation support system to help stroke patients perform some basic operations.