Spatial HuBERT: Self-supervised Spatial Speech Representation Learning for a Single Talker from Multi-channel Audio

TL;DR

Spatial HuBERT is a novel self-supervised model that learns both acoustic and spatial features from multi-channel audio, significantly improving spatial speech tasks in noisy environments and aiding in speech localisation.

Contribution

It introduces Spatial HuBERT, the first self-supervised model to learn spatial speech representations from multi-channel audio for a single speaker, surpassing single-channel methods.

Findings

Outperforms state-of-the-art single-channel speech representations in spatial tasks

Effective in reverberant and noisy environments

Demonstrates utility in speech localisation

Abstract

Self-supervised learning has been used to leverage unlabelled data, improving accuracy and generalisation of speech systems through the training of representation models. While many recent works have sought to produce effective representations across a variety of acoustic domains, languages, modalities and even simultaneous speakers, these studies have all been limited to single-channel audio recordings. This paper presents Spatial HuBERT, a self-supervised speech representation model that learns both acoustic and spatial information pertaining to a single speaker in a potentially noisy environment by using multi-channel audio inputs. Spatial HuBERT learns representations that outperform state-of-the-art single-channel speech representations on a variety of spatial downstream tasks, particularly in reverberant and noisy environments. We also demonstrate the utility of the representations learned by Spatial HuBERT on a speech localisation downstream task. Along with this paper, we publicly release a new dataset of 100 000 simulated first-order ambisonics room impulse responses.