Sequential Multi-Frame Neural Beamforming for Speech Separation and Enhancement

TL;DR

This paper presents a sequential neural beamforming approach combining spectral separation and spatial beamforming, significantly improving speech separation and enhancement performance in reverberant conditions.

Contribution

It introduces a novel multi-frame neural beamforming method with advanced covariance modeling and contextual frames, achieving state-of-the-art results in speech separation.

Findings

Average 2.75 dB SI-SNR improvement

14.2% reduction in speech recognition error

Effective separation of real recordings in LibriCSS

Abstract

This work introduces sequential neural beamforming, which alternates between neural network based spectral separation and beamforming based spatial separation. Our neural networks for separation use an advanced convolutional architecture trained with a novel stabilized signal-to-noise ratio loss function. For beamforming, we explore multiple ways of computing time-varying covariance matrices, including factorizing the spatial covariance into a time-varying amplitude component and a time-invariant spatial component, as well as using block-based techniques. In addition, we introduce a multi-frame beamforming method which improves the results significantly by adding contextual frames to the beamforming formulations. We extensively evaluate and analyze the effects of window size, block size, and multi-frame context size for these methods. Our best method utilizes a sequence of three neural separation and multi-frame time-invariant spatial beamforming stages, and demonstrates an average improvement of 2.75 dB in scale-invariant signal-to-noise ratio and 14.2% absolute reduction in a comparative speech recognition metric across four challenging reverberant speech enhancement and separation tasks. We also use our three-speaker separation model to separate real recordings in the LibriCSS evaluation set into non-overlapping tracks, and achieve a better word error rate as compared to a baseline mask based beamformer.