An Adapter based Multi-label Pre-training for Speech Separation and Enhancement

TL;DR

This paper introduces an adapter-based multi-label pre-training method for HuBERT that enhances speech separation, enhancement, and recognition performance by integrating separation and denoising objectives with minimal additional parameters.

Contribution

It proposes a novel adapter-based architecture for HuBERT that maintains ASR performance while significantly improving speech separation and enhancement tasks.

Findings

Improved performance on speech separation and enhancement tasks.

Maintained or enhanced ASR accuracy.

Faster pre-training with minimal parameter increase.

Abstract

In recent years, self-supervised learning (SSL) has achieved tremendous success in various speech tasks due to its power to extract representations from massive unlabeled data. However, compared with tasks such as speech recognition (ASR), the improvements from SSL representation in speech separation (SS) and enhancement (SE) are considerably smaller. Based on HuBERT, this work investigates improving the SSL model for SS and SE. We first update HuBERT's masked speech prediction (MSP) objective by integrating the separation and denoising terms, resulting in a multiple pseudo label pre-training scheme, which significantly improves HuBERT's performance on SS and SE but degrades the performance on ASR. To maintain its performance gain on ASR, we further propose an adapter-based architecture for HuBERT's Transformer encoder, where only a few parameters of each layer are adjusted to the multiple pseudo label MSP while other parameters remain frozen as default HuBERT. Experimental results show that our proposed adapter-based multiple pseudo label HuBERT yield consistent and significant performance improvements on SE, SS, and ASR tasks, with a faster pre-training speed, at only marginal parameters increase.