Generalized Spatio-Temporal RNN Beamformer for Target Speech Separation

TL;DR

This paper introduces a novel spatio-temporal RNN-based beamformer for target speech separation, which learns beamforming weights directly from covariance matrices, improving speech quality and noise reduction over traditional methods.

Contribution

It proposes a new RNN-based framework for beamforming that automatically learns weights from covariance matrices, with improved variants using layer normalization.

Findings

GRNN-BF outperforms prior methods in PESQ, SNR, and WER.

Layer normalization enhances beamformer performance.

The approach effectively reduces residual noise in separated speech.

Abstract

Although the conventional mask-based minimum variance distortionless response (MVDR) could reduce the non-linear distortion, the residual noise level of the MVDR separated speech is still high. In this paper, we propose a spatio-temporal recurrent neural network based beamformer (RNN-BF) for target speech separation. This new beamforming framework directly learns the beamforming weights from the estimated speech and noise spatial covariance matrices. Leveraging on the temporal modeling capability of RNNs, the RNN-BF could automatically accumulate the statistics of the speech and noise covariance matrices to learn the frame-level beamforming weights in a recursive way. An RNN-based generalized eigenvalue (RNN-GEV) beamformer and a more generalized RNN beamformer (GRNN-BF) are proposed. We further improve the RNN-GEV and the GRNN-BF by using layer normalization to replace the commonly used mask normalization on the covariance matrices. The proposed GRNN-BF obtains better performance against prior arts in terms of speech quality (PESQ), speech-to-noise ratio (SNR) and word error rate (WER).