WaveCycleGAN: Synthetic-to-natural speech waveform conversion using cycle-consistent adversarial networks

TL;DR

WaveCycleGAN introduces a cycle-consistent adversarial network approach to directly convert synthetic speech waveforms into natural-sounding speech, improving naturalness without relying on vocoders or explicit waveform assumptions.

Contribution

This paper presents a novel waveform-level conversion method using cycle-consistent adversarial networks that enhances speech naturalness and reduces over-smoothing effects.

Findings

Significantly improves speech naturalness in synthetic-to-natural conversion.

Reduces over-smoothing effects in acoustic features.

Operates directly on waveforms without explicit waveform assumptions.

Abstract

We propose a learning-based filter that allows us to directly modify a synthetic speech waveform into a natural speech waveform. Speech-processing systems using a vocoder framework such as statistical parametric speech synthesis and voice conversion are convenient especially for a limited number of data because it is possible to represent and process interpretable acoustic features over a compact space, such as the fundamental frequency (F0) and mel-cepstrum. However, a well-known problem that leads to the quality degradation of generated speech is an over-smoothing effect that eliminates some detailed structure of generated/converted acoustic features. To address this issue, we propose a synthetic-to-natural speech waveform conversion technique that uses cycle-consistent adversarial networks and which does not require any explicit assumption about speech waveform in adversarial learning. In contrast to current techniques, since our modification is performed at the waveform level, we expect that the proposed method will also make it possible to generate `vocoder-less' sounding speech even if the input speech is synthesized using a vocoder framework. The experimental results demonstrate that our proposed method can 1) alleviate the over-smoothing effect of the acoustic features despite the direct modification method used for the waveform and 2) greatly improve the naturalness of the generated speech sounds.