Ultrasound-based Articulatory-to-Acoustic Mapping with WaveGlow Speech Synthesis

TL;DR

This paper presents a novel ultrasound-to-speech conversion system using a flow-based neural vocoder (WaveGlow) that produces more natural speech by directly synthesizing from ultrasound tongue images and mel-spectrograms, improving over traditional vocoders.

Contribution

The study introduces a WaveGlow-based neural vocoder for articulatory-to-acoustic mapping, enabling direct synthesis from ultrasound images with enhanced speech naturalness.

Findings

WaveGlow produces significantly more natural speech than baseline.

Including F0 in mel-spectrogram simplifies the process.

The system outperforms traditional vocoding methods.

Abstract

For articulatory-to-acoustic mapping using deep neural networks, typically spectral and excitation parameters of vocoders have been used as the training targets. However, vocoding often results in buzzy and muffled final speech quality. Therefore, in this paper on ultrasound-based articulatory-to-acoustic conversion, we use a flow-based neural vocoder (WaveGlow) pre-trained on a large amount of English and Hungarian speech data. The inputs of the convolutional neural network are ultrasound tongue images. The training target is the 80-dimensional mel-spectrogram, which results in a finer detailed spectral representation than the previously used 25-dimensional Mel-Generalized Cepstrum. From the output of the ultrasound-to-mel-spectrogram prediction, WaveGlow inference results in synthesized speech. We compare the proposed WaveGlow-based system with a continuous vocoder which does not use strict voiced/unvoiced decision when predicting F0. The results demonstrate that during the articulatory-to-acoustic mapping experiments, the WaveGlow neural vocoder produces significantly more natural synthesized speech than the baseline system. Besides, the advantage of WaveGlow is that F0 is included in the mel-spectrogram representation, and it is not necessary to predict the excitation separately.