Handling Trade-Offs in Speech Separation with Sparsely-Gated Mixture of Experts

TL;DR

This paper introduces a sparsely-gated mixture-of-experts architecture for monaural speech separation, effectively balancing model size, separation quality, and computational cost, especially in overlapped speech scenarios.

Contribution

The paper proposes a novel sparsely-gated MoE architecture that improves speech separation performance while reducing artifacts and maintaining low computational overhead.

Findings

Achieves superior separation with less distortion

Maintains low computational cost with marginal runtime increase

Effective on both simulated and real recordings

Abstract

Employing a monaural speech separation (SS) model as a front-end for automatic speech recognition (ASR) involves balancing two kinds of trade-offs. First, while a larger model improves the SS performance, it also requires a higher computational cost. Second, an SS model that is more optimized for handling overlapped speech is likely to introduce more processing artifacts in non-overlapped-speech regions. In this paper, we address these trade-offs with a sparsely-gated mixture-of-experts (MoE) architecture. Comprehensive evaluation results obtained using both simulated and real meeting recordings show that our proposed sparsely-gated MoE SS model achieves superior separation capabilities with less speech distortion, while involving only a marginal run-time cost increase.