Adaptive Slimming for Scalable and Efficient Speech Enhancement

TL;DR

This paper presents a dynamic slimming approach for speech enhancement models that adaptively adjusts computational resources based on input, achieving similar or better quality with fewer calculations.

Contribution

It introduces a scalable, input-adaptive slimming technique for DEMUCS, enabling efficient resource use without sacrificing speech quality.

Findings

Achieves 29% reduction in MACs while maintaining quality.

Operates at 10% capacity on average, matching static 25% models.

Demonstrates Pareto-optimal performance across utilization factors.

Abstract

Speech enhancement (SE) enables robust speech recognition, real-time communication, hearing aids, and other applications where speech quality is crucial. However, deploying such systems on resource-constrained devices involves choosing a static trade-off between performance and computational efficiency. In this paper, we introduce dynamic slimming to DEMUCS, a popular SE architecture, making it scalable and input-adaptive. Slimming lets the model operate at different utilization factors (UF), each corresponding to a different performance/efficiency trade-off, effectively mimicking multiple model sizes without the extra storage costs. In addition, a router subnet, trained end-to-end with the backbone, determines the optimal UF for the current input. Thus, the system saves resources by adaptively selecting smaller UFs when additional complexity is unnecessary. We show that our solution is Pareto-optimal against individual UFs, confirming the benefits of dynamic routing. When training the proposed dynamically-slimmable model to use 10% of its capacity on average, we obtain the same or better speech quality as the equivalent static 25% utilization while reducing MACs by 29%.