MIMO Self-attentive RNN Beamformer for Multi-speaker Speech Separation

TL;DR

This paper introduces a self-attentive RNN beamformer that leverages temporal and spatial self-attention modules to enhance multi-speaker speech separation, improving ASR accuracy and speech quality over previous methods.

Contribution

It proposes a novel self-attentive RNN beamformer with temporal and spatial attention modules, and a multi-channel MIMO model for more efficient multi-speaker speech separation.

Findings

Improved ASR accuracy compared to prior methods.

Enhanced speech quality as measured by PESQ.

Better modeling of covariance matrices through self-attention.

Abstract

Recently, our proposed recurrent neural network (RNN) based all deep learning minimum variance distortionless response (ADL-MVDR) beamformer method yielded superior performance over the conventional MVDR by replacing the matrix inversion and eigenvalue decomposition with two recurrent neural networks. In this work, we present a self-attentive RNN beamformer to further improve our previous RNN-based beamformer by leveraging on the powerful modeling capability of self-attention. Temporal-spatial self-attention module is proposed to better learn the beamforming weights from the speech and noise spatial covariance matrices. The temporal self-attention module could help RNN to learn global statistics of covariance matrices. The spatial self-attention module is designed to attend on the cross-channel correlation in the covariance matrices. Furthermore, a multi-channel input with multi-speaker directional features and multi-speaker speech separation outputs (MIMO) model is developed to improve the inference efficiency. The evaluations demonstrate that our proposed MIMO self-attentive RNN beamformer improves both the automatic speech recognition (ASR) accuracy and the perceptual estimation of speech quality (PESQ) against prior arts.