Learning Robust Spatial Representations from Binaural Audio through Feature Distillation

TL;DR

This paper introduces a self-supervised pretraining method using feature distillation to learn robust spatial representations from binaural audio, improving DoA estimation in noisy and reverberant environments.

Contribution

It proposes a novel pretraining framework that leverages clean binaural features for self-supervised learning of spatial audio representations, enhancing downstream DoA estimation performance.

Findings

Pretrained models outperform fully supervised models in noisy environments.

The approach improves robustness of spatial representations in reverberant conditions.

The method surpasses classic signal processing techniques in DoA estimation accuracy.

Abstract

Recently, deep representation learning has shown strong performance in multiple audio tasks. However, its use for learning spatial representations from multichannel audio is underexplored. We investigate the use of a pretraining stage based on feature distillation to learn a robust spatial representation of binaural speech without the need for data labels. In this framework, spatial features are computed from clean binaural speech samples to form prediction labels. These clean features are then predicted from corresponding augmented speech using a neural network. After pretraining, we throw away the spatial feature predictor and use the learned encoder weights to initialize a DoA estimation model which we fine-tune for DoA estimation. Our experiments demonstrate that the pretrained models show improved performance in noisy and reverberant environments after fine-tuning for direction-of-arrival estimation, when compared to fully supervised models and classic signal processing methods.

Tables

Table 1. Table 1: Summary of evaluated models. SFD denotes pretraining with the proposed spatial feature distillation framework.

Model	Architecture	# Param.
GCCPHAT-Argmax	GCCPHAT-AvgPool-Argmax	0
GCC-DNN	GCC-Conformer-Linear	415.56 k
GCCPHAT-DNN	GCCPHAT-Conformer-Linear	415.56 k
STFT-DNN	STFT-Conformer-Linear	545 k
SFD-GCC	STFT-Conformer-Linear	545 k
SFD-GCC-PHAT	STFT-Conformer-Linear	545 k
SFD-CPSPhase	STFT-Conformer-Linear	545 k
SFD-ILD+IPD	STFT-Conformer-Linear	545 k

Equations

$$x^m[t]=\sum _{i=0}^{N_{\mathrm{s}}-1}({\mathbf{h}}^{i,m}\ast {\mathbf{s}}^i)[t]+v[t{]}^m,$$

$${\mathbf{X}}^m[n]=\mathrm{STFT}({\mathbf{x}}^m)[n],$$

$$\varphi [n]=\arcsin \left(\frac{c\cdot {\mathrm{ITD}}^{m_1,m_2}[n]}{\Vert {\mathbf{d}}^{m_1,m_2}\Vert }\right),$$

$${\mathrm{CPS}}^{m_1,m_2}[n,f]$$

$$\mathrm{IPD}[n,f]=\arg ({\mathrm{CPS}}^{m_1,m_2}[n,f])$$

$${\mathrm{ITD}}^{m_1,m_2}[n,f]=\frac{{\mathrm{IPD}}^{m_1,m_2}[n,f]}{\omega [f]},$$

$${\mathrm{GCC}}^{m_1,m_2}[n,\tau ]=\mathrm{IDFT}({\alpha }^{m_1,m_2}{\mathbf{CPS}}^{m_1,m_2}[n])[\tau ],$$

$${\alpha }_{\mathrm{PHAT}}^{m_1,m_2}[n,f]={\left|{\mathrm{CPS}}^{m_1,m_2}[n,f]\right|}^{-1},$$

$${\mathrm{ITD}}^{m_1,m_2}[n]=\frac{{\mathrm{argmax}}_{\tau }{(\mathrm{GCC}}_{\mathrm{PHAT}}^{m_1,m_2}[n,\tau ])}{F_{\mathrm{s}}},$$

$${\hat{\mathbf{X}}}^m[n]$$

$$\hat{\mathbf{X}}[n]$$

$$\mathbf{z}[n]=\mathrm{SFE}(X^{\prime m_1}[n],X^{\prime m_2}[n]),$$

$$\mathbf{y}[n]=\mathrm{Linear}\left(\mathrm{Conformer}\left({\hat{\mathbf{X}}}^{\prime }[n]\right)\right),$$

$${\mathcal{L}}_{\mathrm{MSE}}=\frac{1}{N}\sum _{n=0}^{N-1}{\Vert \mathbf{y}[n]-\mathbf{z}[n]\Vert }_2^2,$$

Full text

1 Introduction

Through evolution, animals, including humans, have developed the ability to extract spatial information from audio signals, allowing them to determine the direction-of-arrival (DoA) of a sound source, even in adverse conditions with noise and reverberation. This ability is based on differences in interaural time difference (ITD) and level difference (ILD), along with cues from the shapes of the head and ears [1]. Many signal processing algorithms try to replicate this ability for applications such as hearing aids, robots, and teleconferencing. These methods generally try to extract the interaural phase difference (IPD) which is directly related to the ITD. Popular algiorithms include the generalized cross-correlation (GCC) and its phase transform version (GCC-PHAT) [2, 3, 4, 5, 6], steered response power (SRP) [7] and multiple signal classification (MUSIC) [8]. These classic methods usually perform well in simple environments; however, their performance deteriorates significantly in a noisy and reverberant environment [6].

Data-driven models like Gaussian mixture models (GMMs) [9] and deep neural networks (DNNs) [10, 11, 12, 13] have also been proposed for DoA estimation. DNNs, in particular, have shown to excel in noisy and reverberant environments [10, 6]. In the DNN setup, one or more hand-crafted spatial features, such as GCC-PHAT or IPD, are usually passed to the model as input features. Alternatively, some DNN-based methods use short-time Fourier transform (STFT) features directly, instead of relying on hand-crafted spatial features.

The idea of enhancing hand-crafted spatial features has also been explored [14, 15]. In [14], the authors propose to enhance IPD features, using oracle IPDs as the target. This approach was found to perform better than the direct estimation of DoAs from IPDs. Most recently, IPDNet [16] was proposed, which predicts narrowband direct path IPDs from STFT input features. Here, the target is formed by theoretically deriving narrowband IPDs for the DoA label, instead of directly predicting the DoA.

The area of representation learning has shown that DNNs can learn highly performative representations of audio data. Especially, self-supervised learning (SSL) has gained traction in audio signal processing, as it enables learning from unlabelled data [17, 18, 19, 20, 21, 22]. SSL describes a paradigm in which pseudo-labels are automatically derived from unlabelled data. The pseudo-lables are then used for training a neural network on a pretext task, with the goal of learning a good data representation. Examples of pretext tasks include predicting future values [18], masked prediction [17, 20, 19, 22] and contrastive learning [19, 23].

Although much effort has been put towards learning robust audio representations [21, 24, 25], the focus has largely been on single-channel audio, and spatial representation learning remains underexplored. Some self-supervised/unsupervised approaches have been proposed [26, 27], although they mainly focus on speech separation, rather than spatial learning, and do not operate on binaural audio. Recently, SSLSAR [28] was proposed, using cross-channel reconstruction as the pretext task in binaural setups. However, this work does not address noisy, reverberant data or causal frame-level prediction, essential for many applications, such as DoA estimation in hearing aids.

This paper investigates the use of spatial features as spatial representation learning targets and proposes a pretraining framework catered towards learning robust causal frame-level spatial features without the need for DoA labels by predicting spatial features computed from clean speech from noisy speech, similar to WavLM [21]. To evaluate the learned representation, we fine-tune the pretrained models to predict the direction of arrival of a speech source.

Due to the lack of readily available large binaural data sets with annotated direction of arrivals, we create a benchmark data set based on Librispeech [29] and LibriLight [30] data, which is binauralized using recorded head-related transfer functions (HRTF) from the ARI HRTF database [31]. We systematically evaluate the DoA estimation performance in different noisy environments and at various noise levels to assess the robustness of the learned representations.

The key contributions of this paper are:

• •

A novel learning approach for learning robust causal spatial features from binaural speech without the use of spatial annotations.

• •

A comparison of different spatial features for use as spatial representation learning targets.

• •

An extensive evaluation of various binaural DoA estimation approaches in clean and noisy conditions, demonstrating the effectiveness of our pretraining framework for learning a robust spatial representation.

The code and data set111https://github.com/HolgerBovbjerg/spatial_feature_distillation used for model training and generation of the results presented in this paper are publicly available.

2 Direction-of-arrival Estimation

For a multichannel audio signal, the observed signal can be described by (1):

[TABLE]

where $\bm{x}^{m}[t]$ represents the $t$’th sample of the observed signal at microphone channel $m$, $N_{\mathrm{s}}$ is the number of sources, $\bm{h}^{i,m}$ is the impulse response related to source $i$ and microphone $m$, $\bm{s}^{i}$ is the $i$’th source signal and $v^{m}[t]$ represents additive noise.

Signal analysis is usually performed in a framewise manner in the frequency domain by applying STFT to the observed signal [7]. This results in STFT frames:

[TABLE]

where $\mathbf{X}^{m}[n]$ is the sequence of STFT frame representations of the framed observed signal $\bm{x}^{m}$ for the $n$’th frame.

The relevant information for estimating the direction of arrival of a source signal $\bm{s}$, is contained in the ITD, ILD and other cues arrising due to $\bm{h}$ in (1).

2.1 Model-based DoA estimation

Classic model-based methods generally assume that a single source signal is dominant, and rely on estimating the ITD which in combination with the array geometry can be used to estimate the DoA. Assuming a binaural microphone setup, the DoA can be estimated from the ITD as:

[TABLE]

where $\mathbf{d}^{\mathrm{m_{1},m_{2}}}$ is a vector representing the distance between the spatial positions of microphones $m_{1}$ and $m_{2}$, $c$ is the speed of sound, and $\bm{\phi}[n]$ is the estimated DoA for the $n$th frame relative to the normal vector of $\mathbf{d}^{\mathrm{m_{1},m_{2}}}$.

The ITD is directly related to the IPD which can be found by computing the cross-power spectrum (CPS):

[TABLE]

where $\mathrm{CPS}^{m_{1},m_{2}}[n,f]$ denotes the CPS coefficient of the $n$’th frame at frequency index $f$ between microphones $m_{1}$ and $m_{2}$, and $\overline{X}$ denotes the complex conjugate.

The IPD is found as the phase of the CPS:

[TABLE]

The ITD can then be estimated from the IPD as:

[TABLE]

where $\omega[f]$ is the angular frequency corresponding to frequency bin $f$.

Another way to estimate the ITD, is to transform the CPS to time domain by applying an inverse discrete Fourier transform (IDFT) to obtain the generalized cross-correlation (GCC) [2, 3, 7, 5]:

[TABLE]

where $\tau$ denotes a specific delay between $m_{1}$ and $m_{2}$ in number of samples, and $\alpha$ is an optional frequency-domain weighting function.

The weighting function $\alpha$ is commonly chosen as the phase transform (PHAT), which has been shown to be more robust to noise. PHAT-weighting whitens the spectrum by normalizing each CPS frequency component in (4) by its magnitude, such that only the phase information is kept:

[TABLE]

where $\alpha_{\mathrm{PHAT}}^{m_{1},m_{2}}$ is the PHAT-weighting.

To estimate the ITD, one can simply find the value of $\tau$ that maximizes (7), and divide by the sampling frequency to convert it to seconds:

[TABLE]

where $F_{\mathrm{s}}$ is the sampling frequency.

2.2 DNN-based DoA estimation

While model-based features like GCC and IPD have been shown to be good estimators of the DoA, they are prone to noise and reverberation. Some studies have investigated using GCC or IPD as the input features to a DNN to improve the DoA prediction performance [32, 33], combining the model-based and a data-driven approaches.

Another approach is to rely on the DNN to learn relevant spatial information, such as channel correlations, instead of using hand-crafted spatial features. A common setup is to use STFT features as input to the DNN [34, 28]. Here, STFT features are first computed for each channel, as in (2). The real and imaginary parts are then concatenated for each channel to form a real-valued input feature for each channel. Finally, the channelwise features are concatenated to form a single feature vector, such that

[TABLE]

where $\mathbf{\hat{X}}[n]$ denotes a single feature vector containing real and imaginary parts of the STFT, $\mathbf{X}^{m}_{\mathbb{R}}$ and $\mathbf{X}^{m}_{\mathbb{I}}$, from microphone channels $m_{1}$ and $m_{2}$ corresponding to the $n$’th frame.

Using (11) as input, allows the DNN more freedom to potentially learn a spatial representation that is more robust to noise and reverberation than GCC and IPD features. However, the downside is that achieving good performance requires vast amounts of labelled training data which can be difficult to obtain.

The DoA estimation problem can be posed as a linear regression task, e.g., using a Haversine loss [35] between the predicted and true DoA. However, most DoA estimation DNNs are trained using cross-entropy, by quantizing the possible range of DoAs to a fixed set of $k$ directions, $\{\theta_{1},\theta_{2},\dots,\theta_{k}\}$, assigning the class label as the angle closest to the true DoA. In this work, we follow the classification approach using a cross-entropy loss.

3 Feature Distillation Framework

In the proposed setup, a neural network, consisting of an encoder and a feature predictor, is first trained to predict spatial features computed from clean speech, based on noisy input speech. Whereas feature enhancement methods [14, 15, 16] use the predicted features to directly predict DoAs, we discard the feature predictor and instead use the learned encoder representation for downstream DoA prediction, similar to many SSL methods [20, 21, 28].

As described in Section 2, classic spatial features are effective for DoA estimation in relatively clean conditions. DNN based approaches can improve estimation in noisy and revereberant environments, however, with the need for training data with DoA labels.

Inspired by the spatial feature enhancement methods, we propose the use of spatial features as representation learning targets in a feature distillation setup. We hypothesize that classic spatial features serve as effective training targets for learning robust spatial features, removing the need for DoA labels. Specifically, we propose a pretraining task where a DNN predicts spatial features, computed from a clean microphone signal, from a noisy and reverberant input, similar to some noise-robust SSL methods [21, 24, 25]. We refer to this framework as Spatial Feature Distillation (SFD). An illustration of the proposed framework is shown in Figure 1.

As depicted in Figure 1, spatial features are computed from a clean non-reverberant binaural signal, as in (7), yielding target vectors:

[TABLE]

where $\mathbf{z}[n]$, is the target vector for the $n$’th frame and $\mathbf{SFE}$ is a spatial feature extractor. Here we use either GCC/GCC-PHAT, as described in (7), or IPD from (5) concatenated with the + ILD.

In the other branch, a corresponding noisy and reverberant signal is passed through an STFT feature extractor, as in (11). The STFT features are then encoded by a causal Conformer encoder [36] followed by a prediction module, producing predicted spatial features:

[TABLE]

where $\mathbf{y}[n]$ is the prediction for the $n$’th frame , and $\mathbf{\hat{X}}[n]$ denote concatenated STFT features as in (11).

The model is trained to minimize the MSE between $\mathbf{y}[n]$ from (13) and $\mathbf{z}[n]$ from (12) such that:

[TABLE]

where $\mathcal{L}$ denotes the loss, $n$ is the frame index and $N$ is the number of frames per sample.

After pretraining, we use the pretrained model weights to initialize a DoA prediction model, consisting of a conformer encoder with a linear classification head, as shown in Figure 1.

4 Experimental Setup

In this study, we consider a scenario with a single static speech source in a noisy environment and a binaural microphone setup. The method can be extended to incorporate more microphones and accommodate more complex environments; however, we will leave this for future work.

4.1 BinauralLibrispeech Dataset

Currently, there are no large publicly available binaural speech datasets, especially those with source direction and microphone geometry annotations. Therefore, we simulate binaural data for our experiments, similar to [37]. While this may limit real-world generalization, our focus is on exploring learning a spatial audio representation from large unlabelled binaural datasets, enabling its use when available - whether simulated or real.

Our binaural dataset is based on LibriSpeech [29] and LibriLight [30] utterances which we convolve the left and right HRTFs from the ARI HRTF dataset [31] to generate binaural signals. To add reverb we use simulated room impulse responses from SLR28 [38].

To prevent HRTF leakage, we split the ARI HRTF dataset into training, validation, and test sets, with an 80:10:10 split of subjects. HRTFs are sourced from over 250 subjects, recorded with a spatial resolution of $5\text{\,}\frac{\mathrm{\SIUnitSymbolDegree}}{}$. The training data is based on LibriLight 10h, 1h and 10min splits to simulate varying amounts of labelled training data. Validation and test data are generated from LibriSpeech dev-clean and test-clean, respectively.

Binaural utterances are generated by sampling an utterance, a subject from the ARI HRTF dataset, and an RIR from the RIR dataset and an azimuth angle $\phi\sim\mathcal{U}($-90\text{,}\frac{\mathrm{\SIUnitSymbolDegree}}{}$,$90\text{,}\frac{\mathrm{\SIUnitSymbolDegree}}{}$)$ is sampled while fixing the elevation angle at $90\text{\,}\frac{\mathrm{\SIUnitSymbolDegree}}{}$ (horizontal plane). The mono speech signal is convolved with the RIR and the left-right HRTFs are then applied to generate the final binaural signal, which is saved as a .flac file. The microphone array geometry and DoA are stored as metadata, enabling DoA prediction. The static generated binaural dataset is used for supervised training/fine-tuning and benchmarking of models. During pretraining binaural utterances are generated in an online fashion following the same procedure, using all $960\text{\,}\frac{\mathrm{h}}{}$ of speech from the librispeech training data as the basis.

4.2 Pretraining Setup

The general pretraining setup is shown in Figure 1. We pretrain a total of four models, each with a different spatial feature extractor, namely GCC, GCC-PHAT, CPS-PHAT phase and a lastly a the concatenation of ILD and IPD features(8) as the pretraining target. Pretraining uses the full $960\text{\,}\frac{\mathrm{h}}{}$ LibriSpeech training data, binauralized in an online fashion, as described in Section 4.1, using the training HRTFs. Noise augmentation is done by adding diffuse environmental noise, generated with anf-generator[39], with an SNR between $-20\text{\,}\frac{\mathrm{dB}}{}20\text{\,}\frac{\mathrm{dB}}{}$. Here, we use the same noise as in [40] consisting of bus, babble, café, mixed, pedestrian, street and speech shaped noise [40]. We add noise at SNRs levels of $-20\text{\,}\frac{\mathrm{dB}}{}20\text{\,}\frac{\mathrm{dB}}{}$ in steps of $5\text{\,}\frac{\mathrm{dB}}{}$.

We follow [28] and extract STFT features as in (11), using a window length of 400, a hop length of 160, and 512 FFT coefficients. The STFT features are passed to a 2-layer causal Conformer encoder with a 64-dimensional embedding, 4 attention heads, and a convolution kernel size of 31, similar to [28], but with causal masking and a smaller embedding dimension for a more lightweight model. A final linear layer predicts the spatial feature targets, which are computed using the same STFT parameters as the encoder input.

We use an MSE loss, defined in (14), and train for 50k steps, using bucket batching with a bucket size of $10\text{\,}\frac{\mathrm{min}}{}$, and save the best model based on validation performance. The learning rate follows a cosine annealing schedule with warm-up restarts, using a maximum learning rate of 1e-3, a minimum of 5e-7, 3k warm-up steps, and 30k steps per cycle. The cycle length scales by 0.9 and the maximum learning rate by 0.5 after each cycle. We use AdamW [41] with $\beta=[0.9,0.98]$ and $\varepsilon=1.e-6$, and a weight decay of 0.01. Pretraining takes approx. $14\text{\,}\frac{\mathrm{h}}{}$ on a single NVIDIA A40 GPU.

4.3 Supervised Fine-tuning

For supervised fine-tuning, we use the same feature extractor and encoder as in pretraining, discarding the feature predictor and adding a linear classification layer for DoA prediction, as shown in Figure 1. We predict DOAs with an angular resolution of $5\text{\,}\frac{\mathrm{\SIUnitSymbolDegree}}{}$. During fine-tuning, we also apply diffuse noise augmentation as in pretraining. The objective is a cross-entropy loss as described in Section 2.2. The same hyperparameters as pretraining are used, as they were found to work well. However, we use a fixed batch size of 8 utterances and adjust the scheduler to have only 10 warm-up steps and 1k steps per cycle due to less training data. Fine-tuning on a single NVIDIA T4 GPU takes approx. $4\text{\,}\frac{\mathrm{h}}{}$ for the $10\text{\,}\frac{\mathrm{h}}{}$ training set.

To evaluate the fine-tuned model, we test on the binaural test-clean dataset under clean and noisy conditions. The noise types are the same as used during training; however, the individual noise clips are separate test clips different from the ones used for training. For all noise types we evaluate with SNRs between $-20\text{\,}\frac{\mathrm{dB}}{}20\text{\,}\frac{\mathrm{dB}}{}$ in steps of $10\text{\,}\frac{\mathrm{dB}}{}$.

4.4 Baselines

We train three DNN-based reference systems using the same general setup as for fine-tuning, using either GCC, GCC-PHAT, or STFT features as the input. This is followed by a 2-layer causal Conformer encoder and a linear classifier, same as our fine-tuned model.

Lastly, we evaluate a classic approach using argmax on GCC-PHAT features as in (3) applying average pooling using a window of 100 frames. This yields a total of six reference baselines. A summary of all DoA prediction setups is shown in Table 1.

5 Results

In the experimental results analysis, we investigate the encoder output after pretraining, and whether the fine-tuned models outperform purely supervised DoA models.

5.1 DoA prediction performance

Table 2 compares the mean angular error (MAE) on the test set, averaged over all noise types, for models trained on the Binaural LibriLight 10h dataset. After fine-tuning for DoA prediction, all pretrained models show significant improvements in test set performance over reference methods. The SFD-CPSPhase model achieves the lowest MAE at all noise levels, outperforming the best supervised model (GCC-PHAT-DNN) by $46.44\text{\,}\frac{\mathrm{\char 37\relax}}{}$ on average. The SFD-GCC-PHAT model achieves a slightly worse average score than SFD-CPSPhase, however, it performs sligthly better for $-10\text{\,}\frac{\mathrm{dB}}{}$ and $-20\text{\,}\frac{\mathrm{dB}}{}$. SFD-GCC and ILD+IPD performs worst of the pretraining targets, suggesting that including ILD information in the pretraining target is detrimental to performance.

When varying the amount of labelled data, Figure 2 shows that pretraining with GCC-PHAT and CPS Phase consistently yields good performance, even with as little as $10\text{\,}\frac{\mathrm{min}}{}$ of labelled data. In contrast, the supervised DNN baselines suffer significantly from data limitations. This underscores the value of our proposed framework in low-data settings for learning robust spatial representations without DoA labels, similar to the use case of SSL features in other speech processing applications [19].

6 Conclusion

We presented a framework method for learning a robust spatial representation from unlabelled binaural audio data by predicting spatial features. Evaluations in different noisy and reverberant environments show that it outperforms both classic and supervised approaches for DoA estimation of a single static speaker, reducing the mean angular error by $33.6\text{\,}\frac{\mathrm{\char 37\relax}}{}$ on average. A limitation of our current work, is the assumption of clean speech being available. Future work will explore scenarios with no available clean speech, as well as handling of moving sources, multiple speakers, and the effects of scaling pretraining data and model size.