Speech waveform synthesis from MFCC sequences with generative adversarial networks

TL;DR

This paper presents a novel approach to synthesize high-quality speech from MFCC sequences by combining neural prediction, spectral conversion, and GAN-based noise modeling, enabling realistic speech reconstruction from limited features.

Contribution

It introduces a new method that enables speech synthesis solely from MFCCs using a combination of neural networks and GANs, which was previously considered infeasible.

Findings

High-quality speech can be reconstructed from MFCCs.

The GAN-based noise model improves naturalness of synthesized speech.

The method outperforms traditional approaches in speech quality metrics.

Abstract

This paper proposes a method for generating speech from filterbank mel frequency cepstral coefficients (MFCC), which are widely used in speech applications, such as ASR, but are generally considered unusable for speech synthesis. First, we predict fundamental frequency and voicing information from MFCCs with an autoregressive recurrent neural net. Second, the spectral envelope information contained in MFCCs is converted to all-pole filters, and a pitch-synchronous excitation model matched to these filters is trained. Finally, we introduce a generative adversarial network -based noise model to add a realistic high-frequency stochastic component to the modeled excitation signal. The results show that high quality speech reconstruction can be obtained, given only MFCC information at test time.