An Audio-textual Diffusion Model For Converting Speech Signals Into Ultrasound Tongue Imaging Data

TL;DR

This paper introduces a novel audio-textual diffusion model that combines speaker-specific acoustic features and universal linguistic information to generate high-quality ultrasound tongue imaging data from speech signals.

Contribution

The proposed model uniquely integrates wav2vec 2.0 and BERT encodings within a diffusion framework for improved UTI data synthesis.

Findings

Generated UTI data has clearer tongue contours.

Model outperforms existing AAI methods.

High-quality data benefits linguistic and clinical analysis.

Abstract

Acoustic-to-articulatory inversion (AAI) is to convert audio into articulator movements, such as ultrasound tongue imaging (UTI) data. An issue of existing AAI methods is only using the personalized acoustic information to derive the general patterns of tongue motions, and thus the quality of generated UTI data is limited. To address this issue, this paper proposes an audio-textual diffusion model for the UTI data generation task. In this model, the inherent acoustic characteristics of individuals related to the tongue motion details are encoded by using wav2vec 2.0, while the ASR transcriptions related to the universality of tongue motions are encoded by using BERT. UTI data are then generated by using a diffusion module. Experimental results showed that the proposed diffusion model could generate high-quality UTI data with clear tongue contour that is crucial for the linguistic analysis and clinical assessment. The project can be found on the website\footnote{https://yangyudong2020.github.io/wav2uti/