BLOOM-Net: Blockwise Optimization for Masking Networks Toward Scalable and Efficient Speech Enhancement

TL;DR

BLOOM-Net introduces a blockwise optimization approach for scalable speech enhancement networks, allowing dynamic adjustment of complexity with minimal performance loss, enabling flexible resource management.

Contribution

It proposes a novel blockwise training method for scalable masking-based speech enhancement networks, improving flexibility and efficiency over traditional end-to-end training.

Findings

Achieves scalability with minimal performance degradation.

Allows dynamic adjustment of run-time complexity.

Maintains low memory and training overhead.

Abstract

In this paper, we present a blockwise optimization method for masking-based networks (BLOOM-Net) for training scalable speech enhancement networks. Here, we design our network with a residual learning scheme and train the internal separator blocks sequentially to obtain a scalable masking-based deep neural network for speech enhancement. Its scalability lets it dynamically adjust the run-time complexity depending on the test time environment. To this end, we modularize our models in that they can flexibly accommodate varying needs for enhancement performance and constraints on the resources, incurring minimal memory or training overhead due to the added scalability. Our experiments on speech enhancement demonstrate that the proposed blockwise optimization method achieves the desired scalability with only a slight performance degradation compared to corresponding models trained end-to-end.