Fish Audio S2 Technical Report

TL;DR

Fish Audio S2 is an open-source text-to-speech system that supports multi-speaker, multi-turn, and instruction-following generation, with a scalable training pipeline and a production-ready inference engine.

Contribution

The paper introduces Fish Audio S2, a novel open-source TTS system with instruction-following capabilities and a comprehensive training pipeline, including multi-stage data processing.

Findings

Achieves an RTF of 0.195 for streaming inference

Supports multi-speaker and multi-turn speech generation

Provides open-source model weights and inference tools

Abstract

We introduce Fish Audio S2, an open-sourced text-to-speech system featuring multi-speaker, multi-turn generation, and, most importantly, instruction-following control via natural-language descriptions. To scale training, we develop a multi-stage training recipe together with a staged data pipeline covering video captioning and speech captioning, voice-quality assessment, and reward modeling. To push the frontier of open-source TTS, we release our model weights, fine-tuning code, and an SGLang-based inference engine. The inference engine is production-ready for streaming, achieving an RTF of 0.195 and a time-to-first-audio below 100 ms.Our code and weights are available on GitHub (https://github.com/fishaudio/fish-speech) and Hugging Face (https://huggingface.co/fishaudio/s2-pro). We highly encourage readers to visit https://fish.audio to try custom voices.