Multimodal Segmentation for Vocal Tract Modeling

TL;DR

This paper introduces a multimodal segmentation method combining MRI and audio to improve vocal tract modeling, significantly increasing labeled data and setting new benchmarks in MRI video segmentation.

Contribution

It presents a novel deep learning segmentation approach that leverages audio to enhance MRI-based vocal tract modeling and releases a large, labeled RT-MRI dataset.

Findings

Achieved a new benchmark in MRI vocal tract segmentation.

Increased publicly available labeled RT-MRI data by over 9 times.

Demonstrated the effectiveness of multimodal data in improving segmentation accuracy.

Abstract

Accurate modeling of the vocal tract is necessary to construct articulatory representations for interpretable speech processing and linguistics. However, vocal tract modeling is challenging because many internal articulators are occluded from external motion capture technologies. Real-time magnetic resonance imaging (RT-MRI) allows measuring precise movements of internal articulators during speech, but annotated datasets of MRI are limited in size due to time-consuming and computationally expensive labeling methods. We first present a deep labeling strategy for the RT-MRI video using a vision-only segmentation approach. We then introduce a multimodal algorithm using audio to improve segmentation of vocal articulators. Together, we set a new benchmark for vocal tract modeling in MRI video segmentation and use this to release labels for a 75-speaker RT-MRI dataset, increasing the amount of labeled public RT-MRI data of the vocal tract by over a factor of 9. The code and dataset labels can be found at \url{rishiraij.github.io/multimodal-mri-avatar/}.