# Two-Stream Joint-Training for Speaker Independent Acoustic-to-Articulatory Inversion

## TL;DR

This paper introduces a two-stream neural network for acoustic-to-articulatory inversion that leverages phoneme and speech features to enhance speaker-independent performance, achieving significant improvements over state-of-the-art methods.

## Contribution

The paper proposes a novel two-stream network incorporating phoneme features for speaker-independent AAI, a first in the field, improving accuracy and correlation.

## Key findings

- Reduces RMSE by 0.18mm compared to SOTA.
- Increases Pearson correlation coefficient by 6%.
- First to use phoneme features for speaker-independent AAI.

## Abstract

Acoustic-to-articulatory inversion (AAI) aims to estimate the parameters of articulators from speech audio. There are two common challenges in AAI, which are the limited data and the unsatisfactory performance in speaker independent scenario. Most current works focus on extracting features directly from speech and ignoring the importance of phoneme information which may limit the performance of AAI. To this end, we propose a novel network called SPN that uses two different streams to carry out the AAI task. Firstly, to improve the performance of speaker-independent experiment, we propose a new phoneme stream network to estimate the articulatory parameters as the phoneme features. To the best of our knowledge, this is the first work that extracts the speaker-independent features from phonemes to improve the performance of AAI. Secondly, in order to better represent the speech information, we train a speech stream network to combine the local features and the global features. Compared with state-of-the-art (SOTA), the proposed method reduces 0.18mm on RMSE and increases 6.0% on Pearson correlation coefficient in the speaker-independent experiment. The code has been released at https://github.com/liujinyu123/AAINetwork-SPN.