Predicting Video features from EEG and Vice versa

TL;DR

This study investigates deep learning models to predict facial and lip video features from EEG signals and vice versa, aiming to synthesize high-quality facial videos from EEG data, with initial promising results on a seven-subject dataset.

Contribution

It introduces a novel approach for cross-modal prediction between EEG signals and facial video features using deep learning, pioneering the synthesis of facial videos from EEG data.

Findings

Model can generate broad facial/video features from EEG.

First step towards high-quality video synthesis from EEG.

Results demonstrated on seven subjects.

Abstract

In this paper we explore predicting facial or lip video features from electroencephalography (EEG) features and predicting EEG features from recorded facial or lip video frames using deep learning models. The subjects were asked to read out loud English sentences shown to them on a computer screen and their simultaneous EEG signals and facial video frames were recorded. Our model was able to generate very broad characteristics of the facial or lip video frame from input EEG features. Our results demonstrate the first step towards synthesizing high quality facial or lip video from recorded EEG features. We demonstrate results for a data set consisting of seven subjects.