Fish-Speech: Leveraging Large Language Models for Advanced Multilingual Text-to-Speech Synthesis

TL;DR

Fish-Speech introduces a novel multilingual TTS framework using large language models and a dual autoregressive architecture to improve naturalness, efficiency, and multilingual support in speech synthesis.

Contribution

The paper presents Fish-Speech, a new TTS system that leverages LLMs for linguistic features and a dual-AR architecture to enhance stability and quality, streamlining multilingual speech synthesis.

Findings

Outperforms baseline models in complex linguistic scenarios

Achieves near 100% codebook utilization with FF-GAN

Enhances multilingual and voice cloning capabilities

Abstract

Text-to-Speech (TTS) systems face ongoing challenges in processing complex linguistic features, handling polyphonic expressions, and producing natural-sounding multilingual speech - capabilities that are crucial for future AI applications. In this paper, we present Fish-Speech, a novel framework that implements a serial fast-slow Dual Autoregressive (Dual-AR) architecture to enhance the stability of Grouped Finite Scalar Vector Quantization (GFSQ) in sequence generation tasks. This architecture improves codebook processing efficiency while maintaining high-fidelity outputs, making it particularly effective for AI interactions and voice cloning. Fish-Speech leverages Large Language Models (LLMs) for linguistic feature extraction, eliminating the need for traditional grapheme-to-phoneme (G2P) conversion and thereby streamlining the synthesis pipeline and enhancing multilingual support. Additionally, we developed FF-GAN through GFSQ to achieve superior compression ratios and near 100\% codebook utilization. Our approach addresses key limitations of current TTS systems while providing a foundation for more sophisticated, context-aware speech synthesis. Experimental results show that Fish-Speech significantly outperforms baseline models in handling complex linguistic scenarios and voice cloning tasks, demonstrating its potential to advance TTS technology in AI applications. The implementation is open source at \href{https://github.com/fishaudio/fish-speech}{https://github.com/fishaudio/fish-speech}.