JoyTTS: LLM-based Spoken Chatbot With Voice Cloning

TL;DR

JoyTTS is an end-to-end spoken chatbot integrating large language models with voice cloning, trained on extensive conversational data, enabling realistic voice synthesis and conversational capabilities.

Contribution

It introduces a novel LLM-based spoken chatbot with voice cloning, built on open-source models and trained on 2000 hours of data, with accessible code for community development.

Findings

Achieves SS score of 0.73 indicating high speaker similarity

Attains a WER of 5.09 demonstrating accurate speech recognition

Provides open-source code and models for further research

Abstract

JoyTTS is an end-to-end spoken chatbot that combines large language models (LLM) with text-to-speech (TTS) technology, featuring voice cloning capabilities. This project is built upon the open-source MiniCPM-o and CosyVoice2 models and trained on 2000 hours of conversational data. We have also provided the complete training code to facilitate further development and optimization by the community. On the testing machine seed-tts-zh, it achieves a SS (speaker similarity) score of 0.73 and a WER (Word Error Rate) of 5.09. The code and models, along with training and inference scripts, are available at https://github.com/jdh-algo/JoyTTS.git.