RNN-based speech synthesis using a continuous sinusoidal model

TL;DR

This paper introduces an RNN-based speech synthesis framework utilizing a continuous sinusoidal model with contF0, MVF, and MGC, achieving natural and intelligible speech comparable to high-quality models.

Contribution

It combines a continuous sinusoidal model with RNNs, specifically Bi-LSTM, for improved speech synthesis, including a novel post-processing step for contF0 refinement.

Findings

Achieved naturalness and intelligibility comparable to high-quality models

Enhanced contF0 estimation through time-warping post-processing

Demonstrated effectiveness with objective and subjective evaluations

Abstract

Recently in statistical parametric speech synthesis, we proposed a continuous sinusoidal model (CSM) using continuous F0 (contF0) in combination with Maximum Voiced Frequency (MVF), which was successfully giving state-of-the-art vocoders performance (e.g. similar to STRAIGHT) in synthesized speech. In this paper, we address the use of sequence-to-sequence modeling with recurrent neural networks (RNNs). Bidirectional long short-term memory (Bi-LSTM) is investigated and applied using our CSM to model contF0, MVF, and Mel-Generalized Cepstrum (MGC) for more natural sounding synthesized speech. For refining the output of the contF0 estimation, post-processing based on time-warping approach is applied to reduce the unwanted voiced component of the unvoiced speech sounds, resulting in an enhanced contF0 track. The overall conclusion is covered by objective evaluation and subjective listening test, showing that the proposed framework provides satisfactory results in terms of naturalness and intelligibility, and is comparable to the high-quality WORLD model based RNNs.