Accelerating High-Fidelity Waveform Generation via Adversarial Flow Matching Optimization

TL;DR

This paper presents PeriodWave-Turbo, a waveform generation model that significantly accelerates high-fidelity speech synthesis using adversarial flow matching, reducing inference steps and improving quality with minimal fine-tuning.

Contribution

It introduces a novel adversarial flow matching optimization method that enhances CFM models, enabling high-quality waveform generation with fewer steps and improved generalization.

Findings

Achieves state-of-the-art PESQ score of 4.454 on LibriTTS

Reduces inference steps from 16 to 2-4

Requires only 1,000 fine-tuning steps for high performance

Abstract

This paper introduces PeriodWave-Turbo, a high-fidelity and high-efficient waveform generation model via adversarial flow matching optimization. Recently, conditional flow matching (CFM) generative models have been successfully adopted for waveform generation tasks, leveraging a single vector field estimation objective for training. Although these models can generate high-fidelity waveform signals, they require significantly more ODE steps compared to GAN-based models, which only need a single generation step. Additionally, the generated samples often lack high-frequency information due to noisy vector field estimation, which fails to ensure high-frequency reproduction. To address this limitation, we enhance pre-trained CFM-based generative models by incorporating a fixed-step generator modification. We utilized reconstruction losses and adversarial feedback to accelerate high-fidelity waveform generation. Through adversarial flow matching optimization, it only requires 1,000 steps of fine-tuning to achieve state-of-the-art performance across various objective metrics. Moreover, we significantly reduce inference speed from 16 steps to 2 or 4 steps. Additionally, by scaling up the backbone of PeriodWave from 29M to 70M parameters for improved generalization, PeriodWave-Turbo achieves unprecedented performance, with a perceptual evaluation of speech quality (PESQ) score of 4.454 on the LibriTTS dataset. Audio samples, source code and checkpoints will be available at https://github.com/sh-lee-prml/PeriodWave.