Combining Masked Language Modeling and Cross-Modal Contrastive Learning for Prosody-Aware TTS

TL;DR

This paper explores multi-stage pretraining for prosody modeling in diffusion-based TTS, combining masked language modeling and cross-modal contrastive learning to improve synthesis quality.

Contribution

It introduces a dual-stream encoder trained with MLM and contrastive learning, revealing insights into balancing phoneme discrimination and prosodic sensitivity.

Findings

Two-stage curriculum improves synthesis quality in TTS

Same-phoneme refinement enhances prosodic retrieval but degrades synthesis

Embedding metrics do not always correlate with generative performance

Abstract

We investigate multi-stage pretraining for prosody modeling in diffusion-based TTS. A speaker-conditioned dual-stream encoder is trained with masked language modeling followed by SigLIP-style cross-modal contrastive learning using mixed-phoneme batches, with an additional same-phoneme refinement stage studied separately. We evaluate intrinsic text-audio retrieval and downstream synthesis in Grad-TTS and a latent diffusion TTS system. The two-stage curriculum (MLM + mixed-phoneme contrastive learning) achieves the best overall synthesis quality in terms of intelligibility, speaker similarity, and perceptual measures. Although same-phoneme refinement improves prosodic retrieval, it reduces phoneme discrimination and degrades synthesis. These findings indicate that improvements in embedding-space metrics do not necessarily translate to better generative performance and highlight the need to balance phoneme discrimination and prosodic sensitivity in TTS pretraining.