Phonetic Enhanced Language Modeling for Text-to-Speech Synthesis

TL;DR

This paper introduces a phonetic enhanced language modeling approach for text-to-speech synthesis, improving robustness and detail by leveraging phonetic representations and non-autoregressive acoustic prediction.

Contribution

It proposes combining phonetic-rich self-supervised representations with non-autoregressive acoustic modeling to enhance TTS performance and robustness.

Findings

Improved speech synthesis quality in objective evaluations.

Enhanced robustness against error propagation.

Subjective tests show higher naturalness of generated speech.

Abstract

Recent language model-based text-to-speech (TTS) frameworks demonstrate scalability and in-context learning capabilities. However, they suffer from robustness issues due to the accumulation of errors in speech unit predictions during autoregressive language modeling. In this paper, we propose a phonetic enhanced language modeling method to improve the performance of TTS models. We leverage self-supervised representations that are phonetically rich as the training target for the autoregressive language model. Subsequently, a non-autoregressive model is employed to predict discrete acoustic codecs that contain fine-grained acoustic details. The TTS model focuses solely on linguistic modeling during autoregressive training, thereby reducing the error propagation that occurs in non-autoregressive training. Both objective and subjective evaluations validate the effectiveness of our proposed method.