MossFormer2: Combining Transformer and RNN-Free Recurrent Network for Enhanced Time-Domain Monaural Speech Separation

TL;DR

MossFormer2 introduces a hybrid model combining transformer and RNN-free recurrent networks to improve monaural speech separation by capturing both long-range and fine-scale dependencies.

Contribution

The paper proposes a novel hybrid architecture integrating a feedforward sequential memory network (FSMN) with MossFormer, enhancing speech separation performance over existing models.

Findings

Outperforms MossFormer and state-of-the-art methods on multiple benchmarks.

Effectively models both long-range dependencies and fine-scale recurrent patterns.

Achieves significant improvements in speech separation accuracy.

Abstract

Our previously proposed MossFormer has achieved promising performance in monaural speech separation. However, it predominantly adopts a self-attention-based MossFormer module, which tends to emphasize longer-range, coarser-scale dependencies, with a deficiency in effectively modelling finer-scale recurrent patterns. In this paper, we introduce a novel hybrid model that provides the capabilities to model both long-range, coarse-scale dependencies and fine-scale recurrent patterns by integrating a recurrent module into the MossFormer framework. Instead of applying the recurrent neural networks (RNNs) that use traditional recurrent connections, we present a recurrent module based on a feedforward sequential memory network (FSMN), which is considered "RNN-free" recurrent network due to the ability to capture recurrent patterns without using recurrent connections. Our recurrent module mainly comprises an enhanced dilated FSMN block by using gated convolutional units (GCU) and dense connections. In addition, a bottleneck layer and an output layer are also added for controlling information flow. The recurrent module relies on linear projections and convolutions for seamless, parallel processing of the entire sequence. The integrated MossFormer2 hybrid model demonstrates remarkable enhancements over MossFormer and surpasses other state-of-the-art methods in WSJ0-2/3mix, Libri2Mix, and WHAM!/WHAMR! benchmarks (https://github.com/modelscope/ClearerVoice-Studio).