VoiceSculptor: Your Voice, Designed By You

TL;DR

VoiceSculptor is an open-source system that enables fine-grained, instruction-based control over speech attributes and high-fidelity voice cloning, advancing reproducible research in controllable TTS.

Contribution

It introduces a unified framework combining instruction-based voice design with high-fidelity voice cloning, supporting iterative refinement and attribute-level editing.

Findings

Achieves state-of-the-art results on InstructTTSEval-Zh.

Supports controllable speaker timbre from natural language descriptions.

Fully open-sourced with code and pretrained models.

Abstract

Despite rapid progress in text-to-speech (TTS), open-source systems still lack truly instruction-following, fine-grained control over core speech attributes (e.g., pitch, speaking rate, age, emotion, and style). We present VoiceSculptor, an open-source unified system that bridges this gap by integrating instruction-based voice design and high-fidelity voice cloning in a single framework. It generates controllable speaker timbre directly from natural-language descriptions, supports iterative refinement via Retrieval-Augmented Generation (RAG), and provides attribute-level edits across multiple dimensions. The designed voice is then rendered into a prompt waveform and fed into a cloning model to enable high-fidelity timbre transfer for downstream speech synthesis. VoiceSculptor achieves open-source state-of-the-art (SOTA) on InstructTTSEval-Zh, and is fully open-sourced, including code and pretrained models, to advance reproducible instruction-controlled TTS research.