Perceptual Loss based Speech Denoising with an ensemble of Audio Pattern Recognition and Self-Supervised Models

TL;DR

This paper introduces PERL, a perceptual ensemble regularization loss for speech denoising, leveraging multiple pre-trained models to improve perceptual quality, and finds that a focused acoustic event model outperforms more complex multi-model approaches.

Contribution

The paper proposes PERL, a novel perceptual loss framework for speech denoising that effectively utilizes pre-trained models, with a simplified approach outperforming complex multi-task learning methods.

Findings

PERL with acoustic event model outperforms state-of-the-art methods.

Multi-task learning with multiple losses faces challenges and underperforms compared to single-loss tuning.

Using all models simultaneously does not improve denoising due to domain mismatch and loss incompatibility.

Abstract

Deep learning based speech denoising still suffers from the challenge of improving perceptual quality of enhanced signals. We introduce a generalized framework called Perceptual Ensemble Regularization Loss (PERL) built on the idea of perceptual losses. Perceptual loss discourages distortion to certain speech properties and we analyze it using six large-scale pre-trained models: speaker classification, acoustic model, speaker embedding, emotion classification, and two self-supervised speech encoders (PASE+, wav2vec 2.0). We first build a strong baseline (w/o PERL) using Conformer Transformer Networks on the popular enhancement benchmark called VCTK-DEMAND. Using auxiliary models one at a time, we find acoustic event and self-supervised model PASE+ to be most effective. Our best model (PERL-AE) only uses acoustic event model (utilizing AudioSet) to outperform state-of-the-art methods on major perceptual metrics. To explore if denoising can leverage full framework, we use all networks but find that our seven-loss formulation suffers from the challenges of Multi-Task Learning. Finally, we report a critical observation that state-of-the-art Multi-Task weight learning methods cannot outperform hand tuning, perhaps due to challenges of domain mismatch and weak complementarity of losses.