RefineGAN: Universally Generating Waveform Better than Ground Truth with Highly Accurate Pitch and Intensity Responses

TL;DR

RefineGAN is a neural vocoder that achieves high-fidelity waveform generation with accurate pitch and intensity, outperforming ground truth in subjective tests and generalizing well across languages and speakers.

Contribution

The paper introduces RefineGAN, a novel GAN-based neural vocoder with a pitch-guided refine architecture and multi-scale spectrogram loss for improved robustness and accuracy.

Findings

Generated audio surpasses ground-truth in subjective quality.

Model maintains performance on unseen languages and speakers.

High-speed full-band audio generation achieved.

Abstract

Most GAN(Generative Adversarial Network)-based approaches towards high-fidelity waveform generation heavily rely on discriminators to improve their performance. However, GAN methods introduce much uncertainty into the generation process and often result in mismatches of pitch and intensity, which is fatal when it comes to sensitive use cases such as singing voice synthesis(SVS). To address this problem, we propose RefineGAN, a high-fidelity neural vocoder focused on the robustness, pitch and intensity accuracy, and high-speed full-band audio generation. We applyed a pitch-guided refine architecture with a multi-scale spectrogram-based loss function to help stabilize the training process and maintain the robustness of the neural vocoder while using the GAN-based training method. Audio generated using this method shows a better performance in subjective tests when compared with the ground-truth audio. This result shows that the fidelity is even improved during the waveform reconstruction by eliminating defects produced by recording procedures. Moreover, it shows that models trained on a specified type of data can perform on totally unseen language and unseen speaker identically well. Generated sample pairs are provided onhttps://timedomain-tech.github.io/refinegan/.