NeuroHeed: Neuro-Steered Speaker Extraction using EEG Signals

TL;DR

NeuroHeed is a novel EEG-guided speaker extraction model that uses brain signals to improve speech separation in noisy environments, enabling real-time processing with high quality and intelligibility.

Contribution

This paper introduces NeuroHeed, a new EEG-based speaker extraction framework with offline and online versions, including a real-time autoregressive speaker encoder for continuous attention tracking.

Findings

Effective extraction of attended speech signals demonstrated

High perceptual quality and intelligibility achieved

Real-time NeuroHeed performs well in two-speaker scenarios

Abstract

Humans possess the remarkable ability to selectively attend to a single speaker amidst competing voices and background noise, known as selective auditory attention. Recent studies in auditory neuroscience indicate a strong correlation between the attended speech signal and the corresponding brain's elicited neuronal activities, which the latter can be measured using affordable and non-intrusive electroencephalography (EEG) devices. In this study, we present NeuroHeed, a speaker extraction model that leverages EEG signals to establish a neuronal attractor which is temporally associated with the speech stimulus, facilitating the extraction of the attended speech signal in a cocktail party scenario. We propose both an offline and an online NeuroHeed, with the latter designed for real-time inference. In the online NeuroHeed, we additionally propose an autoregressive speaker encoder, which accumulates past extracted speech signals for self-enrollment of the attended speaker information into an auditory attractor, that retains the attentional momentum over time. Online NeuroHeed extracts the current window of the speech signals with guidance from both attractors. Experimental results demonstrate that NeuroHeed effectively extracts brain-attended speech signals, achieving high signal quality, excellent perceptual quality, and intelligibility in a two-speaker scenario.