Learning to Inference with Early Exit in the Progressive Speech Enhancement

TL;DR

This paper introduces a stage-wise adaptive inference method with early exit for speech enhancement, allowing for faster processing without sacrificing quality, and proposes an improved model PL-CRN++ that outperforms existing systems.

Contribution

It presents a novel early exit mechanism for progressive speech enhancement and an improved model PL-CRN++ that combines stage recurrent mechanisms and complex spectral mapping.

Findings

The proposed system outperforms state-of-the-art baselines in PESQ, ESTOI, and DNSMOS.

Adjustable thresholds enable control over inference speed and system performance.

Extensive experiments on TIMIT demonstrate the effectiveness of the approach.

Abstract

In real scenarios, it is often necessary and significant to control the inference speed of speech enhancement systems under different conditions. To this end, we propose a stage-wise adaptive inference approach with early exit mechanism for progressive speech enhancement. Specifically, in each stage, once the spectral distance between adjacent stages lowers the empirically preset threshold, the inference will terminate and output the estimation, which can effectively accelerate the inference speed. To further improve the performance of existing speech enhancement systems, PL-CRN++ is proposed, which is an improved version over our preliminary work PL-CRN and combines stage recurrent mechanism and complex spectral mapping. Extensive experiments are conducted on the TIMIT corpus, the results demonstrate the superiority of our system over state-of-the-art baselines in terms of PESQ, ESTOI and DNSMOS. Moreover, by adjusting the threshold, we can easily control the inference efficiency while sustaining the system performance.