Improved Decoding of Attentional Selection in Multi-Talker Environments with Self-Supervised Learned Speech Representation

TL;DR

This study demonstrates that self-supervised learned speech representations significantly improve the accuracy and speed of auditory attention decoding in multi-talker environments, advancing brain-controlled hearing technologies.

Contribution

The paper introduces the use of WavLM's self-supervised speech representations to enhance auditory attention decoding accuracy and efficiency over traditional waveform and spectrogram methods.

Findings

WavLM representations outperform traditional methods in decoding accuracy.

Decoding speed is improved using self-supervised speech features.

Results suggest potential for brain-controlled hearing aids.

Abstract

Auditory attention decoding (AAD) is a technique used to identify and amplify the talker that a listener is focused on in a noisy environment. This is done by comparing the listener's brainwaves to a representation of all the sound sources to find the closest match. The representation is typically the waveform or spectrogram of the sounds. The effectiveness of these representations for AAD is uncertain. In this study, we examined the use of self-supervised learned speech representation in improving the accuracy and speed of AAD. We recorded the brain activity of three subjects using invasive electrocorticography (ECoG) as they listened to two conversations and focused on one. We used WavLM to extract a latent representation of each talker and trained a spatiotemporal filter to map brain activity to intermediate representations of speech. During the evaluation, the reconstructed representation is compared to each speaker's representation to determine the target speaker. Our results indicate that speech representation from WavLM provides better decoding accuracy and speed than the speech envelope and spectrogram. Our findings demonstrate the advantages of self-supervised learned speech representation for auditory attention decoding and pave the way for developing brain-controlled hearable technologies.