FunCodec: A Fundamental, Reproducible and Integrable Open-source Toolkit for Neural Speech Codec

TL;DR

FunCodec is an open-source neural speech codec toolkit that offers reproducible training, easy integration, and state-of-the-art models like SoundStream and Encodec, supporting downstream tasks such as speech recognition.

Contribution

It introduces FunCodec, a unified, reproducible toolkit with pre-trained models and novel frequency-domain codecs, enhancing speech quality and efficiency.

Findings

FunCodec outperforms other toolkits in speech reconstruction quality.

FreqCodec achieves comparable quality with lower complexity.

Pre-trained models are effective for speech recognition and TTS.

Abstract

This paper presents FunCodec, a fundamental neural speech codec toolkit, which is an extension of the open-source speech processing toolkit FunASR. FunCodec provides reproducible training recipes and inference scripts for the latest neural speech codec models, such as SoundStream and Encodec. Thanks to the unified design with FunASR, FunCodec can be easily integrated into downstream tasks, such as speech recognition. Along with FunCodec, pre-trained models are also provided, which can be used for academic or generalized purposes. Based on the toolkit, we further propose the frequency-domain codec models, FreqCodec, which can achieve comparable speech quality with much lower computation and parameter complexity. Experimental results show that, under the same compression ratio, FunCodec can achieve better reconstruction quality compared with other toolkits and released models. We also demonstrate that the pre-trained models are suitable for downstream tasks, including automatic speech recognition and personalized text-to-speech synthesis. This toolkit is publicly available at https://github.com/alibaba-damo-academy/FunCodec.