Brain-computer interface with rapid serial multimodal presentation using artificial facial images and voice

TL;DR

This study introduces a rapid serial multimodal BCI using artificial facial images and voice, demonstrating improved accuracy through audiovisual stimuli, which could enhance gaze-independent BCI systems.

Contribution

The paper presents a novel RSMP BCI integrating artificial facial images and voice stimuli, showing that audiovisual stimuli enhance performance compared to unimodal stimuli.

Findings

Audiovisual stimuli improved BCI performance.

Online accuracy reached 85.7% with audiovisual stimuli.

P300 at Pz contributed to classification accuracy.

Abstract

Electroencephalography (EEG) signals elicited by multimodal stimuli can drive brain-computer interfaces (BCIs), and research has demonstrated that visual and auditory stimuli can be employed simultaneously to improve BCI performance. However, no studies have investigated the effect of multimodal stimuli in rapid serial visual presentation (RSVP) BCIs. In the present study, we propose a rapid serial multimodal presentation (RSMP) BCI that incorporates artificial facial images and artificial voice stimuli. To clarify the effect of audiovisual stimuli on the RSMP BCI, scrambled images and masked sounds were applied instead of visual and auditory stimuli, respectively. Our findings indicated that the audiovisual stimuli improved the performance of the RSMP BCI, and that the P300 at Pz contributed to classification accuracy. Online accuracy of BCI reached 85.7+-11.5%. Taken together, these findings may aid in the development of better gaze-independent BCI systems.