# Hybrid EEG-fNIRS phoneme classification based on imagined and perceived speech

## TL;DR

This study explores using combined brain signals from EEG and fNIRS to decode imagined and perceived speech phonemes, offering potential for speech prosthetics.

## Contribution

The paper introduces a novel multimodal EEG-fNIRS approach for four-class phoneme decoding in both imagined and perceived speech.

## Key findings

- Hybrid EEG-fNIRS classification achieved 77.29% accuracy for imagined speech and 76.05% for perceived speech.
- EEG-based classification performed similarly to the hybrid method, while fNIRS alone yielded lower accuracy.
- The study outperformed previous research in four-class imagined speech classification using EEG features.

## Abstract

Individuals affected by severe motor impairments often have no means of communicating with others. To build an intuitive speech prosthesis, imagined speech brain-computer interface research began to prosper with numerous studies attempting to classify imagined speech from brain signals. While unimodal neuroimaging techniques, such as electroencephalography (EEG) and functional near-infrared spectroscopy (fNIRS) have been widely used, multimodal approaches combining two or more of them remain scarce.

In this study offline phoneme decoding based on hybrid EEG-fNIRS data was performed. Twenty-two right-handed participants performed imagined and perceived speech trials encompassing four phonemes /a/,/i/,/b/ and /k/. Features in the form of power spectral densities and mean hemoglobin concentration changes were extracted from EEG and fNIRS data, respectively. Features were ranked according to the mutual information criterion relative to the target vector, and the optimal number of features to include was determined through optimization via 10-fold cross-validation.

Hybrid classification yielded accuracy scores of 77.29% and 76.05% regarding imagined and perceived speech, respectively. In both conditions, hybrid and EEG-based classification performances did not differ significantly, while fNIRS based phoneme discrimination produced lower accuracies.

This study represents an innovative phoneme decoding attempt based on multimodal EEG-fNIRS data, both in terms of imagined speech and perception. Four-class imagined speech classification was primarily driven by EEG features yet outperformed comparable previous studies.