Non-Causal to Causal SSL-Supported Transfer Learning: Towards a High-Performance Low-Latency Speech Vocoder

TL;DR

This paper develops a low-latency causal speech vocoder by integrating causal convolutions, transfer learning, and SSL-based representation alignment, achieving high-quality speech synthesis comparable to high-delay models.

Contribution

It introduces a novel transfer learning scheme and SSL-based alignment to enhance low-latency causal vocoders, bridging the performance gap with non-causal models.

Findings

Causal vocoder achieves 3.96 PESQ, surpassing the original non-causal BigVGAN.

Transfer learning and SSL alignment significantly improve low-latency vocoder performance.

The proposed model maintains high speech quality with only 21% increased complexity.

Abstract

Recently, BigVGAN has emerged as high-performance speech vocoder. Its sequence-to-sequence-based synthesis, however, prohibits usage in low-latency conversational applications. Our work addresses this shortcoming in three steps. First, we introduce low latency into BigVGAN via implementing causal convolutions, yielding decreased performance. Second, to regain performance, we propose a teacher-student transfer learning scheme to distill the high-delay non-causal BigVGAN into our low-latency causal vocoder. Third, taking advantage of a self-supervised learning (SSL) model, in our case wav2vec 2.0, we align its encoder speech representations extracted from our low-latency causal vocoder to the ground truth ones. In speaker-independent settings, both proposed training schemes notably elevate the performance of our low-latency vocoder, closing up to the original high-delay BigVGAN. At only 21% higher complexity, our best small causal vocoder achieves 3.96 PESQ and 1.25 MCD, excelling even the original small non-causal BigVGAN (3.64 PESQ) by 0.32 PESQ and 0.1 MCD points, respectively.