FLY-TTS: Fast, Lightweight and High-Quality End-to-End Text-to-Speech Synthesis

TL;DR

FLY-TTS is a novel, fast, and lightweight end-to-end text-to-speech system that leverages ConvNeXt blocks, parameter sharing, and pre-trained models to achieve high-quality speech synthesis with significantly reduced latency and model size.

Contribution

The paper introduces FLY-TTS, a new TTS model that combines innovative architectural modifications and model compression techniques for improved speed and efficiency.

Findings

Achieves real-time synthesis at 0.0139 seconds on CPU

Reduces model size by 1.6 times through parameter sharing

Maintains comparable speech quality to strong baselines

Abstract

While recent advances in Text-To-Speech synthesis have yielded remarkable improvements in generating high-quality speech, research on lightweight and fast models is limited. This paper introduces FLY-TTS, a new fast, lightweight and high-quality speech synthesis system based on VITS. Specifically, 1) We replace the decoder with ConvNeXt blocks that generate Fourier spectral coefficients followed by the inverse short-time Fourier transform to synthesize waveforms; 2) To compress the model size, we introduce the grouped parameter-sharing mechanism to the text encoder and flow-based model; 3) We further employ the large pre-trained WavLM model for adversarial training to improve synthesis quality. Experimental results show that our model achieves a real-time factor of 0.0139 on an Intel Core i9 CPU, 8.8x faster than the baseline (0.1221), with a 1.6x parameter compression. Objective and subjective evaluations indicate that FLY-TTS exhibits comparable speech quality to the strong baseline.