Clip-TTS: Contrastive Text-content and Mel-spectrogram, A High-Quality Text-to-Speech Method based on Contextual Semantic Understanding

TL;DR

Clip-TTS introduces a novel TTS approach leveraging CLIP architecture to enhance semantic understanding during encoding, resulting in high-quality, fast-synthesized speech with state-of-the-art MOS scores on multiple datasets.

Contribution

The paper presents a new TTS method that integrates CLIP to connect text semantics with mel-spectrograms, improving speech quality without sacrificing inference speed.

Findings

Achieves state-of-the-art MOS scores on LJSpeech and Baker datasets.

Demonstrates effective multi-emotion speech synthesis.

Maintains fast inference speeds with Transformer architecture.

Abstract

Traditional text-to-speech (TTS) methods primarily focus on establishing a mapping between phonemes and mel-spectrograms. However, during the phoneme encoding stage, there is often a lack of real mel-spectrogram auxiliary information, which results in the encoding process lacking true semantic understanding. At the same time, traditional TTS systems often struggle to balance the inference speed of the model with the quality of the synthesized speech. Methods that generate high-quality synthesized speech tend to have slower inference speeds, while faster inference methods often sacrifice speech quality. In this paper, I propose Clip-TTS, a TTS method based on the Clip architecture. This method uses the Clip framework to establish a connection between text content and real mel-spectrograms during the text encoding stage, enabling the text encoder to directly learn the true semantics of the global context, thereby ensuring the quality of the synthesized speech. In terms of model architecture, I adopt the basic structure of Transformer, which allows Clip-TTS to achieve fast inference speeds. Experimental results show that on the LJSpeech and Baker datasets, the speech generated by Clip-TTS achieves state-of-the-art MOS scores, and it also performs excellently on multi-emotion datasets.Audio samples are available at: https://ltydd1314.github.io/.