MRI2Speech: Speech Synthesis from Articulatory Movements Recorded by Real-time MRI

TL;DR

MRI2Speech introduces a novel multi-modal self-supervised approach for speech synthesis from real-time MRI data, significantly improving intelligibility and generalization to unseen speakers by predicting text and synthesizing speech without relying on noisy ground-truth speech.

Contribution

The paper adapts AV-HuBERT for MRI-based text prediction and introduces a flow-based duration predictor, enabling high-quality speech synthesis from rtMRI with better speaker generalization.

Findings

Achieves 15.18% WER on USC-TIMIT corpus, outperforming previous methods.

Demonstrates robustness to missing articulator information through masking experiments.

Generalizes well to unseen speakers and different articulator conditions.

Abstract

Previous real-time MRI (rtMRI)-based speech synthesis models depend heavily on noisy ground-truth speech. Applying loss directly over ground truth mel-spectrograms entangles speech content with MRI noise, resulting in poor intelligibility. We introduce a novel approach that adapts the multi-modal self-supervised AV-HuBERT model for text prediction from rtMRI and incorporates a new flow-based duration predictor for speaker-specific alignment. The predicted text and durations are then used by a speech decoder to synthesize aligned speech in any novel voice. We conduct thorough experiments on two datasets and demonstrate our method's generalization ability to unseen speakers. We assess our framework's performance by masking parts of the rtMRI video to evaluate the impact of different articulators on text prediction. Our method achieves a $15.18\%$ Word Error Rate (WER) on the USC-TIMIT MRI corpus, marking a huge improvement over the current state-of-the-art. Speech samples are available at https://mri2speech.github.io/MRI2Speech/