Acoustic-to-articulatory Inversion based on Speech Decomposition and Auxiliary Feature

TL;DR

This paper introduces a novel approach for acoustic-to-articulatory inversion that uses speech decomposition and auxiliary features, significantly improving speaker-independent performance and accuracy over existing methods.

Contribution

The study proposes a pre-trained speech decomposition network and an auxiliary feature network to enhance speaker-independent AAI performance, addressing limitations of previous audio-only methods.

Findings

Reduces average RMSE by 0.29 in speaker-independent case

Increases correlation coefficient by 5.0% in speaker-independent case

Outperforms state-of-the-art methods using only audio features

Abstract

Acoustic-to-articulatory inversion (AAI) is to obtain the movement of articulators from speech signals. Until now, achieving a speaker-independent AAI remains a challenge given the limited data. Besides, most current works only use audio speech as input, causing an inevitable performance bottleneck. To solve these problems, firstly, we pre-train a speech decomposition network to decompose audio speech into speaker embedding and content embedding as the new personalized speech features to adapt to the speaker-independent case. Secondly, to further improve the AAI, we propose a novel auxiliary feature network to estimate the lip auxiliary features from the above personalized speech features. Experimental results on three public datasets show that, compared with the state-of-the-art only using the audio speech feature, the proposed method reduces the average RMSE by 0.25 and increases the average correlation coefficient by 2.0% in the speaker-dependent case. More importantly, the average RMSE decreases by 0.29 and the average correlation coefficient increases by 5.0% in the speaker-independent case.