# Ultrasound-based Silent Speech Interface Built on a Continuous Vocoder

## TL;DR

This paper introduces a continuous vocoder approach for silent speech interfaces using ultrasound tongue images, improving pitch prediction and speech naturalness over traditional methods.

## Contribution

It presents a novel continuous F0 tracker and vocoder that enhance speech synthesis quality in ultrasound-based silent speech interfaces.

## Key findings

- Lower F0 prediction error with continuous F0
- Slightly more natural synthesized speech
- Improved articulatory-to-acoustic mapping

## Abstract

Recently it was shown that within the Silent Speech Interface (SSI) field, the prediction of F0 is possible from Ultrasound Tongue Images (UTI) as the articulatory input, using Deep Neural Networks for articulatory-to-acoustic mapping. Moreover, text-to-speech synthesizers were shown to produce higher quality speech when using a continuous pitch estimate, which takes non-zero pitch values even when voicing is not present. Therefore, in this paper on UTI-based SSI, we use a simple continuous F0 tracker which does not apply a strict voiced / unvoiced decision. Continuous vocoder parameters (ContF0, Maximum Voiced Frequency and Mel-Generalized Cepstrum) are predicted using a convolutional neural network, with UTI as input. The results demonstrate that during the articulatory-to-acoustic mapping experiments, the continuous F0 is predicted with lower error, and the continuous vocoder produces slightly more natural synthesized speech than the baseline vocoder using standard discontinuous F0.