A comparative study of estimating articulatory movements from phoneme sequences and acoustic features

TL;DR

This study compares methods for estimating speech articulatory movements from acoustic signals and phoneme sequences, revealing that linguistic information alone can predict articulatory motion with high accuracy, especially when combining multiple input types.

Contribution

It demonstrates that phoneme sequences alone can effectively estimate articulatory movements, and combining acoustic and phoneme data improves prediction accuracy.

Findings

Attention network with phoneme sequences achieves high correlation (0.81)

Estimation from acoustic signals yields a correlation of 0.85

Combining acoustic and phoneme data increases correlation to 0.88

Abstract

Unlike phoneme sequences, movements of speech articulators (lips, tongue, jaw, velum) and the resultant acoustic signal are known to encode not only the linguistic message but also carry para-linguistic information. While several works exist for estimating articulatory movement from acoustic signals, little is known to what extent articulatory movements can be predicted only from linguistic information, i.e., phoneme sequence. In this work, we estimate articulatory movements from three different input representations: R1) acoustic signal, R2) phoneme sequence, R3) phoneme sequence with timing information. While an attention network is used for estimating articulatory movement in the case of R2, BLSTM network is used for R1 and R3. Experiments with ten subjects' acoustic-articulatory data reveal that the estimation techniques achieve an average correlation coefficient of 0.85, 0.81, and 0.81 in the case of R1, R2, and R3 respectively. This indicates that attention network, although uses only phoneme sequence (R2) without any timing information, results in an estimation performance similar to that using rich acoustic signal (R1), suggesting that articulatory motion is primarily driven by the linguistic message. The correlation coefficient is further improved to 0.88 when R1 and R3 are used together for estimating articulatory movements.