Generative Speech Foundation Model Pretraining for High-Quality Speech Extraction and Restoration

TL;DR

This paper introduces a generative speech foundation model that operates directly on Fourier coefficients, enabling high-quality speech restoration without vocoders, outperforming existing methods across multiple tasks.

Contribution

The paper presents a novel generative pretraining approach for speech restoration that eliminates reliance on vocoders and achieves superior results on various tasks.

Findings

Outperforms strong baselines in speech denoising, bandwidth extension, and artifact removal.

Achieves state-of-the-art results in target speaker extraction, surpassing SSL-based systems.

Operates directly on complex Fourier coefficients, simplifying synthesis and improving quality.

Abstract

This paper proposes a generative pretraining foundation model for high-quality speech restoration tasks. By directly operating on complex-valued short-time Fourier transform coefficients, our model does not rely on any vocoders for time-domain signal reconstruction. As a result, our model simplifies the synthesis process and removes the quality upper-bound introduced by any mel-spectrogram vocoder compared to prior work SpeechFlow. The proposed method is evaluated on multiple speech restoration tasks, including speech denoising, bandwidth extension, codec artifact removal, and target speaker extraction. In all scenarios, finetuning our pretrained model results in superior performance over strong baselines. Notably, in the target speaker extraction task, our model outperforms existing systems, including those leveraging SSL-pretrained encoders like WavLM. The code and the pretrained checkpoints are publicly available in the NVIDIA NeMo framework.