Speaker consistency loss and step-wise optimization for semi-supervised joint training of TTS and ASR using unpaired text data

TL;DR

This paper introduces a novel semi-supervised joint training method for TTS and ASR that uses speaker consistency loss and step-wise optimization to improve speaker characteristic preservation and model robustness.

Contribution

It proposes a speaker consistency loss and step-wise optimization approach to enhance semi-supervised joint TTS and ASR training with unpaired text data.

Findings

Improved speaker characteristic preservation in synthesized speech.

Enhanced ASR accuracy through better cycle-consistency training.

Effective training strategy with step-wise parameter freezing.

Abstract

In this paper, we investigate the semi-supervised joint training of text to speech (TTS) and automatic speech recognition (ASR), where a small amount of paired data and a large amount of unpaired text data are available. Conventional studies form a cycle called the TTS-ASR pipeline, where the multispeaker TTS model synthesizes speech from text with a reference speech and the ASR model reconstructs the text from the synthesized speech, after which both models are trained with a cycle-consistency loss. However, the synthesized speech does not reflect the speaker characteristics of the reference speech and the synthesized speech becomes overly easy for the ASR model to recognize after training. This not only decreases the TTS model quality but also limits the ASR model improvement. To solve this problem, we propose improving the cycleconsistency-based training with a speaker consistency loss and step-wise optimization. The speaker consistency loss brings the speaker characteristics of the synthesized speech closer to that of the reference speech. In the step-wise optimization, we first freeze the parameter of the TTS model before both models are trained to avoid over-adaptation of the TTS model to the ASR model. Experimental results demonstrate the efficacy of the proposed method.