Multi-channel Multi-frame ADL-MVDR for Target Speech Separation

TL;DR

This paper introduces a multi-channel, multi-frame deep learning based MVDR approach for target speech separation that effectively reduces distortions and improves ASR performance, outperforming existing methods.

Contribution

The paper presents a novel MCMF ADL-MVDR system that extends previous work to better handle distortions and utilize spatio-temporal correlations for speech separation.

Findings

Outperforms state-of-the-art methods on Mandarin audio-visual corpus

Reduces residual noise and distortion in separated speech

Improves automatic speech recognition accuracy

Abstract

Many purely neural network based speech separation approaches have been proposed to improve objective assessment scores, but they often introduce nonlinear distortions that are harmful to modern automatic speech recognition (ASR) systems. Minimum variance distortionless response (MVDR) filters are often adopted to remove nonlinear distortions, however, conventional neural mask-based MVDR systems still result in relatively high levels of residual noise. Moreover, the matrix inverse involved in the MVDR solution is sometimes numerically unstable during joint training with neural networks. In this study, we propose a multi-channel multi-frame (MCMF) all deep learning (ADL)-MVDR approach for target speech separation, which extends our preliminary multi-channel ADL-MVDR approach. The proposed MCMF ADL-MVDR system addresses linear and nonlinear distortions. Spatio-temporal cross correlations are also fully utilized in the proposed approach. The proposed systems are evaluated using a Mandarin audio-visual corpus and are compared with several state-of-the-art approaches. Experimental results demonstrate the superiority of our proposed systems under different scenarios and across several objective evaluation metrics, including ASR performance.