Continuous Speech Recognition using EEG and Video

TL;DR

This study explores the integration of EEG signals into a deep learning model to enhance continuous visual speech recognition accuracy.

Contribution

It introduces a novel approach combining EEG features with a CTC-based ASR model for improved speech recognition.

Findings

EEG features improve recognition accuracy

Enhanced performance over baseline models

Demonstrated feasibility of EEG integration in ASR

Abstract

In this paper we investigate whether electroencephalography (EEG) features can be used to improve the performance of continuous visual speech recognition systems. We implemented a connectionist temporal classification (CTC) based end-to-end automatic speech recognition (ASR) model for performing recognition. Our results demonstrate that EEG features are helpful in enhancing the performance of continuous visual speech recognition systems.