Complete reconstruction of the tongue contour through acoustic to articulatory inversion using real-time MRI data

TL;DR

This paper presents a novel deep learning approach using real-time MRI data to accurately reconstruct the entire tongue contour from speech signals, advancing acoustic-to-articulatory inversion techniques.

Contribution

It introduces a new method leveraging high-quality MRI data and explores various neural architectures to improve full tongue contour reconstruction from speech.

Findings

Median accuracy of 2.21 mm in tongue contour reconstruction

Effective use of Bi-MSTM architecture with autoencoder

Demonstrates feasibility of full tongue tracking from acoustic data

Abstract

Acoustic articulatory inversion is a major processing challenge, with a wide range of applications from speech synthesis to feedback systems for language learning and rehabilitation. In recent years, deep learning methods have been applied to the inversion of less than a dozen geometrical positions corresponding to sensors glued to easily accessible articulators. It is therefore impossible to know the shape of the whole tongue from root to tip. In this work, we use high-quality real-time MRI data to track the contour of the tongue. The data used to drive the inversion are therefore the unstructured speech signal and the tongue contours. Several architectures relying on a Bi-MSTM including or not an autoencoder to reduce the dimensionality of the latent space, using or not the phonetic segmentation have been explored. The results show that the tongue contour can be recovered with a median accuracy of 2.21 mm (or 1.37 pixel) taking a context of 1 MFCC frame (static, delta and double-delta cepstral features).